Error installing problemtools on OSX

[arknave]

Tried to install problemtools on a relatively clean OSX instance. Ran into the following errors:

Install Log

$ pip install --user git+https://github.com/kattis/problemtools
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting git+https://github.com/kattis/problemtools
  Cloning https://github.com/kattis/problemtools to /private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm
Collecting PyYAML (from problemtools==1.20190331)
  Downloading https://files.pythonhosted.org/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz (274kB)
    100% |████████████████████████████████| 276kB 4.1MB/s
Collecting plasTeX (from problemtools==1.20190331)
Building wheels for collected packages: problemtools, PyYAML
  Building wheel for problemtools (setup.py) ... error
  Complete output from command /usr/local/opt/python@2/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-wheel-dNICA4 --python-tag cp27:
  readlink: illegal option -- f
  usage: readlink [-n] [file ...]
  running bdist_wheel
  running build
  running build_support
  Running command: make -C support install DESTDIR=/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/build/lib/problemtools/support
  cd checktestdata && ./bootstrap
  + aclocal -I m4
  + autoconf
  + ./configure --disable-parsergeneration
  checking whether configure should try to set CXXFLAGS... yes
  checking whether configure should try to set LDFLAGS... yes
  checking for g++... g++
  checking whether the C++ compiler works... yes
  checking for C++ compiler default output file name... a.out
  checking for suffix of executables...
  checking whether we are cross compiling... no
  checking for suffix of object files... o
  checking whether we are using the GNU C++ compiler... yes
  checking whether g++ accepts -g... yes
  checking whether C++ compiler accepts -Wall... yes
  checking whether C++ compiler accepts -fstack-protector... yes
  checking whether C++ compiler accepts -fPIE... yes
  checking whether C++ compiler accepts -D_FORTIFY_SOURCE=2... yes
  checking whether the linker accepts -fPIE... yes
  checking whether the linker accepts -pie... yes
  checking whether the linker accepts -Wl,-z,relro... no
  checking whether the linker accepts -Wl,-z,now... no
  checking whether we are using the GNU C++ compiler... (cached) yes
  checking whether g++ accepts -g... (cached) yes
  checking for gcc... gcc
  checking whether we are using the GNU C compiler... yes
  checking whether gcc accepts -g... yes
  checking for gcc option to accept ISO C89... none needed
  checking how to run the C++ preprocessor... g++ -E
  checking how to run the C preprocessor... gcc -E
  checking whether g++ supports C++11 features by default... no
  checking whether g++ supports C++11 features with -std=c++11... yes
  checking for proper std::regex implementation... yes
  checking for grep that handles long lines and -e... /usr/bin/grep
  checking for egrep... /usr/bin/grep -E
  checking for ANSI C header files... yes
  checking for sys/types.h... yes
  checking for sys/stat.h... yes
  checking for stdlib.h... yes
  checking for string.h... yes
  checking for memory.h... yes
  checking for strings.h... yes
  checking for inttypes.h... yes
  checking for stdint.h... yes
  checking for unistd.h... yes
  checking gmpxx.h usability... yes
  checking gmpxx.h presence... yes
  checking for gmpxx.h... yes
  checking for abs in -lgmpxx... yes
  checking for libboost regex support... checking build system type... x86_64-apple-darwin18.0.0
  checking host system type... x86_64-apple-darwin18.0.0
  checking for boostlib >= 1.33.1... yes
  configure: parser generation disabled
  configure: creating ./config.status
  config.status: creating config.mk
  /Library/Developer/CommandLineTools/usr/bin/make -C checktestdata; /Library/Developer/CommandLineTools/usr/bin/make -C default_validator; /Library/Developer/CommandLineTools/usr/bin/make -C interactive;
  g++ -std=c++11 -g -O2 -Wall -fstack-protector -fPIE -D_FORTIFY_SOURCE=2 -std=c++11 -DVERSION="\"20190415\"" -I/usr/local/include  -c -o libchecktestdata.o libchecktestdata.cc
  libchecktestdata.cc:1100:22: error: calling a private constructor of class 'std::__1::__wrap_iter<const char *>'
                  if ( !regex_search((string::const_iterator)&data[datanr],
                                     ^
  /Library/Developer/CommandLineTools/usr/include/c++/v1/iterator:1420:31: note: declared private here
      _LIBCPP_INLINE_VISIBILITY __wrap_iter(iterator_type __x) _NOEXCEPT_DEBUG : __i(__x) {}
                                ^
  1 error generated.
  make[1]: *** [libchecktestdata.o] Error 1
  g++ -O3 -o default_validator default_validator.cc
  g++ -O3 -Wall -o interactive interactive.cc
  interactive.cc:194:5: error: use of undeclared identifier 'pipe2'
          if(pipe2(fd, O_CLOEXEC)) {
             ^
  interactive.cc:199:19: error: use of undeclared identifier 'F_SETPIPE_SZ'
          if (fcntl(fd[0], F_SETPIPE_SZ, PIPE_SIZE) == -1) {
                           ^
  2 errors generated.
  make[1]: *** [interactive] Error 1
  make: *** [all] Error 2
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py", line 94, in <module>
      'build': build
    File "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
      dist.run_commands()
    File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
      self.run_command(cmd)
    File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python2.7/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py", line 44, in run
      self.run_command('build_support')
    File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py", line 29, in run
      subprocess.check_call(command)
    File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 190, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['make', '-C', 'support', 'install', 'DESTDIR=/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/build/lib/problemtools/support']' returned non-zero exit status 2

  ----------------------------------------
  Failed building wheel for problemtools
  Running setup.py clean for problemtools
  Building wheel for PyYAML (setup.py) ... done
  Stored in directory: /Users/arnav/Library/Caches/pip/wheels/ad/56/bc/1522f864feb2a358ea6f1a92b4798d69ac783a28e80567a18b
Successfully built PyYAML
Failed to build problemtools
Installing collected packages: PyYAML, plasTeX, problemtools
  Running setup.py install for problemtools ... error
    Complete output from command /usr/local/opt/python@2/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-record-SIuwOU/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    readlink: illegal option -- f
    usage: readlink [-n] [file ...]
    running install
    running build
    running build_support
    Running command: make -C support install DESTDIR=/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/build/lib/problemtools/support
    /Library/Developer/CommandLineTools/usr/bin/make -C checktestdata; /Library/Developer/CommandLineTools/usr/bin/make -C default_validator; /Library/Developer/CommandLineTools/usr/bin/make -C interactive;
    g++ -std=c++11 -g -O2 -Wall -fstack-protector -fPIE -D_FORTIFY_SOURCE=2 -std=c++11 -DVERSION="\"20190415\"" -I/usr/local/include  -c -o libchecktestdata.o libchecktestdata.cc
    libchecktestdata.cc:1100:22: error: calling a private constructor of class 'std::__1::__wrap_iter<const char *>'
                    if ( !regex_search((string::const_iterator)&data[datanr],
                                       ^
    /Library/Developer/CommandLineTools/usr/include/c++/v1/iterator:1420:31: note: declared private here
        _LIBCPP_INLINE_VISIBILITY __wrap_iter(iterator_type __x) _NOEXCEPT_DEBUG : __i(__x) {}
                                  ^
    1 error generated.
    make[1]: *** [libchecktestdata.o] Error 1
    make[1]: Nothing to be done for `build'.
    g++ -O3 -Wall -o interactive interactive.cc
    interactive.cc:194:5: error: use of undeclared identifier 'pipe2'
            if(pipe2(fd, O_CLOEXEC)) {
               ^
    interactive.cc:199:19: error: use of undeclared identifier 'F_SETPIPE_SZ'
            if (fcntl(fd[0], F_SETPIPE_SZ, PIPE_SIZE) == -1) {
                             ^
    2 errors generated.
    make[1]: *** [interactive] Error 1
    make: *** [all] Error 2
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py", line 94, in <module>
        'build': build
      File "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py", line 563, in run
        self.run_command('build')
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
        self.distribution.run_command(command)
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py", line 44, in run
        self.run_command('build_support')
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
        self.distribution.run_command(command)
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py", line 29, in run
        subprocess.check_call(command)
      File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 190, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['make', '-C', 'support', 'install', 'DESTDIR=/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/build/lib/problemtools/support']' returned non-zero exit status 2

    ----------------------------------------
Command "/usr/local/opt/python@2/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-record-SIuwOU/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /private/var/folders/nz/vrjnrxh14fj8j526s1f_hgc40000gn/T/pip-req-build-6nf7Cm/

Looks like the issue is in interactive.cc: BSD based systems don't have pipe2(). I'm not really familiar with UNIX programming, but it looks like we're using pipe2() to set the FD_CLOEXEC flag on the pipe. Is there some reason we can't set it with fcntl() to make this more cross-platform?

EDIT: as always, the man page answers my question. Looks like this is a bad race condition :( More digging turned up this rust issue, but it looks like they just gave up on OSX support.

[simonlindholm]

Using fcntl should be perfectly fine; interactive.cc does not use multithreading so there are no race conditions to worry about. Are you able to make that change locally, check that it works, and send a PR for it? Ideally ./bin/verifyproblem.sh examples/guess/ would succeed without errors (but there may be more problems with running on MacOS, I don't know).

[arknave]

Sure, I'll give that a try in an hour or two. Thanks for the fast response!

[arknave]

$ cd support/interactive
$ make
g++ -O3 -Wall -o interactive interactive.cc
interactive.cc:201:19: error: use of undeclared identifier 'F_SETPIPE_SZ'
        if (fcntl(fd[0], F_SETPIPE_SZ, PIPE_SIZE) == -1) {

I surrender. Going to just try and hack an OSX build without this dependency for now.

[simonlindholm]

FWIW it'd be fine to skip that call on MacOS, it's just a hack for making #113 come up less often in practice. (You could revert 0d38a8b, which seems to be the culprit for both that and the pipe2 call.)

[arknave]

Now checktestdata fails to compiile - I have bison, but not bisonc++. I'll look into setting that up later.

[arknave]

Gave up, using Docker.