Start task fails due to Python dependency issue

[mmduyzend]

Recently started getting errors when the start task executes the following line from install_python_dependencies() in node_scripts/setup_host.sh:

    pipenv run pip install --upgrade pip setuptools wheel

This is preventing us from creating working new clusters. Consider pinning to a working version rather than using --upgrade to fix this issue and avoid it happening again.

Here's the tail of startup/stdout.txt from a node with the start task failure:

Installing python dependencies
Creating a virtualenv for this project...
Pipfile: /mnt/batch/tasks/startup/wd/.aztk-env/Pipfile
Using /usr/bin/python3.5m (3.5.2) to create virtualenv...
Running virtualenv with interpreter /usr/bin/python3.5m
Using base prefix '/usr'
New python executable in /mnt/batch/tasks/startup/wd/.aztk-env/.venv/bin/python3.5m
Also creating executable in /mnt/batch/tasks/startup/wd/.aztk-env/.venv/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /mnt/batch/tasks/startup/wd/.aztk-env/.venv
Installing dependencies from Pipfile.lock (060574)...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pipenv/cli.py", line 435, in install
    selective_upgrade=selective_upgrade,
  File "/usr/local/lib/python3.5/dist-packages/pipenv/core.py", line 1943, in do_install
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.5/dist-packages/pipenv/core.py", line 1322, in do_init
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.5/dist-packages/pipenv/core.py", line 807, in do_install_dependencies
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.5/dist-packages/pipenv/core.py", line 1375, in pip_install
    package_name.split('--hash')[0].split('--trusted-host')[0]
  File "/usr/local/lib/python3.5/dist-packages/pipenv/vendor/requirementslib/models/requirements.py", line 704, in from_line
    line, extras = _strip_extras(line)
TypeError: 'module' object is not callable

[jafreck]

I am currently investigating this. I was able to reproduce the issue. The cause of the issue is an incompatibility between the latest pip and pipenv: pypa/pipenv#2924

You are correct, this should have been pinned along with all other packages installed on the node in the Pipfile.lock.

[jafreck]

For tracking purposes, this also needs to be pinned:

aztk/aztk/node_scripts/setup_host.sh

Line 145 in 4a60c8a

python3 -m pip install pipenv

[jafreck]

This has been fixed with the latest release (0.9.1). If you still experience issues, please let me know either on this issue or a new one. Thanks for reporting this.