Unable to install Cloudmapper

[paulwakeford]

Hi,

macOS 10.13.6 and following the install docs I get:

....
Collecting pandas==0.22.0 (from -r requirements.txt (line 5))
Using cached https://files.pythonhosted.org/packages/08/01/803834bc8a4e708aedebb133095a88a4dad9f45bbaf5ad777d2bea543c7e/pandas-0.22.0.tar.gz
Could not find a version that satisfies the requirement numpy==1.9.3 (from versions: 1.14.5, 1.15.0rc2, 1.15.0)
No matching distribution found for numpy==1.9.3

Installing with

pip install numpy==1.9.3

.. just leads me down the path of esoteric build errors...

numpy/random/mtrand/mtrand.c:34512:13: error: no member named 'exc_type' in 'struct _ts'
tstate->exc_type = local_type;
~~~~~~ ^
numpy/random/mtrand/mtrand.c:34513:13: error: no member named 'exc_value' in 'struct _ts'; did you mean 'curexc_value'?
tstate->exc_value = local_value;

.. and so on. Trying to change requirements.txt to use the latest pandas package gets further but fails compiling pyproj:

#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
_proj.c:5152:17: warning: 'PyObject_AsWriteBuffer' is deprecated [-Wdeprecated-declarations]
  __pyx_t_1 = ((PyObject_AsWriteBuffer(__pyx_v_lats2, (&__pyx_v_distdat), (&__pyx_v_buflend)) != 0) != 0);
                ^
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/abstract.h:503:40: note: 'PyObject_AsWriteBuffer' has been explicitly marked deprecated here
                                       Py_DEPRECATED(3.0);
                                       ^
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pyport.h:493:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
_proj.c:7421:21: error: no member named 'exc_type' in 'struct _ts'
    *type = tstate->exc_type;
            ~~~~~~  ^
_proj.c:7422:22: error: no member named 'exc_value' in 'struct _ts'; did you mean 'curexc_value'?
    *value = tstate->exc_value;
                     ^~~~~~~~~
                     curexc_value

Any ideas?

Thanks,

Paul

[0xdabbad00]

I just tried a fresh setup after seeing your issue, and didn't have any issues. I'm on macOS 10.13.5. Are you working out of a virtualenv environment? You should have done:

python3 -m venv ./venv
source venv/bin/activate

As part of the initialization steps. That should avoid any package version issues that might conflict with other packages on your system.

[paulwakeford]

Yep I did that. Full log:

PWMBP:~ pwakeford$ cd Projects/
PWMBP:Projects pwakeford$ ls
Test1
PWMBP:Projects pwakeford$ mkdir cm4
PWMBP:Projects pwakeford$ cd cm4
PWMBP:cm4 pwakeford$ # clone the repo
PWMBP:cm4 pwakeford$ git clone [email protected]:duo-labs/cloudmapper.git
Cloning into 'cloudmapper'...
remote: Counting objects: 1338, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 1338 (delta 24), reused 33 (delta 18), pack-reused 1293
Receiving objects: 100% (1338/1338), 4.39 MiB | 588.00 KiB/s, done.
Resolving deltas: 100% (611/611), done.
PWMBP:cm4 pwakeford$ # Install pre-reqs for pyjq
PWMBP:cm4 pwakeford$ brew install autoconf automake libtool jq awscli python3
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core, homebrew/cask).
==> Updated Formulae
bear dependency-check guile libxkbcommon quicktype thors-serializer
ckan firebase-cli kubectx mafft sbt webpack
composer geos libgit2 openapi-generator sqlmap
corsixth goenv librealsense phpunit sslyze

Warning: autoconf 2.69 is already installed and up-to-date
To reinstall 2.69, run brew reinstall autoconf
Warning: automake 1.16.1 is already installed and up-to-date
To reinstall 1.16.1, run brew reinstall automake
Warning: libtool 2.4.6_1 is already installed and up-to-date
To reinstall 2.4.6_1, run brew reinstall libtool
Warning: jq 1.5_3 is already installed and up-to-date
To reinstall 1.5_3, run brew reinstall jq
Warning: awscli 1.15.70 is already installed, it's just not linked
You can use brew link awscli to link this version.
Warning: python 3.7.0 is already installed and up-to-date
To reinstall 3.7.0, run brew reinstall python
PWMBP:cm4 pwakeford$ cd cloudmapper/
PWMBP:cloudmapper pwakeford$ python3 -m venv ./venv
PWMBP:cloudmapper pwakeford$ source venv/bin/activate
(venv) PWMBP:cloudmapper pwakeford$ pip install -r requirements.txt
Collecting pyjq==2.1.0 (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/ec/c1/b5fcba48cb715b88b5ad619d1769204411383e2a275dfddda2d2fa8f99dd/pyjq-2.1.0.tar.gz
Collecting netaddr==0.7.19 (from -r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/ba/97/ce14451a9fd7bdb5a397abf99b24a1a6bb7a1a440b019bebd2e9a0dbec74/netaddr-0.7.19-py2.py3-none-any.whl
Collecting six==1.10.0 (from -r requirements.txt (line 3))
Using cached https://files.pythonhosted.org/packages/c8/0a/b6723e1bc4c516cb687841499455a8505b44607ab535be01091c0f24f079/six-1.10.0-py2.py3-none-any.whl
Collecting boto3==1.5.32 (from -r requirements.txt (line 4))
Using cached https://files.pythonhosted.org/packages/22/8b/b120349f2684341fc08b389cb9fca1761a0cd1444b9550ad0ef91073a8d9/boto3-1.5.32-py2.py3-none-any.whl
Collecting pandas==0.22.0 (from -r requirements.txt (line 5))
Using cached https://files.pythonhosted.org/packages/08/01/803834bc8a4e708aedebb133095a88a4dad9f45bbaf5ad777d2bea543c7e/pandas-0.22.0.tar.gz
Could not find a version that satisfies the requirement numpy==1.9.3 (from versions: 1.14.5, 1.15.0rc2, 1.15.0)
No matching distribution found for numpy==1.9.3
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(venv) PWMBP:cloudmapper pwakeford$

[0xdabbad00]

Looks like it might be this issue: pandas-dev/pandas#20697
I'm running Python 3.6.4 and pip 9.0.1. Looks like within the virtualenv, you can run pip install python --upgrade to update your Python version.

[rdpickard]

The most current version of Python from Homebrew is 3.7. This version seems to break a lot package installation. If you step down to 3.6.5 all of the packages build and install without errors.

To step down

brew switch python 3.6.5

I had the same problems you saw with python 3.7.

[ms4720]

this may be a better solution for venvs:

/usr/local/Cellar/python3/3.6.5_1/bin/python3 -m venv venv

[paulwakeford]

Installing and switching to 3.6.5 resolved this - thanks everyone.

[camilosantana]

i ran into this same issue. and was able to workaround with...

brew install pyenv
pyenv install 3.6.5

cat >> ~/.bash_profile <<EOF

# reference: http://mattseymour.net/blog/2016/03/brew-installing-specific-python-version/
export PYENV_ROOT=/usr/local/var/pyenv
if which pyenv > /dev/null; then eval "$(pyenv init -)"; fi

EOF

source ~/.bash_profile

brew install autoconf automake libtool jq awscli
/usr/local/var/pyenv/versions/3.6.5/bin/python3.6 -m venv ./venv
source ./venv/bin/activate
pip install -r requirements.txt

@0xdabbad00 perhaps the macOS instructions would be best updated. would it be useful to submit a PR with updated instructions for new macos machines running latest that need to downgrade python in order to run?

[0xdabbad00]

Is it possible to step down from within the virtualenv? I want to avoid messing with brew or the python installed on the host outside of the virtualenv.

[camilosantana]

my python versions seem untouched. from a new terminal ...

$ python --version
Python 2.7.15
$ python3 --version
Python 3.7.0
$ /usr/local/var/pyenv/versions/3.6.5/bin/python3.6 --version   # <-- obviously
Python 3.6.5

my understanding of python ecosystem is trivial at best but i seem to have not mucked my machine about here. if you wanna skip the bash profile stuff, that's fine. i don't think it's needed in the end.

[0xdabbad00]

PR #160 (which I haven't merged yet) is for using pipenv, which seems to support specifying an exact python version. So using that PR could resolve this issue along with better pinning libraries to use. I'll investigate this weekend.

[camilosantana]

right on. i'll note in my internal docs and watch PR #160

thanks!

[0xdabbad00]

Need to reopen this until the issue of needing to downgrade python is resolved.

[0xdabbad00]

This should be resolved now by using pipenv and Python 3.7 via #187