Multiple sources

[RomanSteinberg]

Issue description

Using multiple source in Pipfile leads to installation failure.

Expected result

Pipenv should install packages respecting filled index.

Actual result

Package installation failed...

Steps to replicate

Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu117"
verify_ssl = true
name = "downloadpytorch"

[packages]
"pyannote.audio" = "*"

[dev-packages]

[requires]
python_version = "3.10"

Commands:

mkdir .venv
pipenv install

The result is very long. It's tail looks like this:

...
8edc8a3239350f59fe7', 'sha256:8288d7cd28f8119b07dd49b7230d6b4562f9b61ee9a4ab02221060d21136be80', 'sha256:2b0738fb871812722a0ac2154be1f049c6223b9f6f22eec352996b69775b36d4', 'sha256:82aa6264b36c50acfb2424ad5ca537a2060ab6de158a5bd2a72a032cc75b9eb8', 'sha256:662e6016409828ee910f5d9602a2729a8a57d74b163c89a837de3fea050c7582', 'sha256:59723a029760079b7d991a401386390c4be5bfec1e7dd83e25a6a0881859e716', 'sha256:f4e2d08f07a3d7d3e12549052eb5ad3eab1c349c53ac51c209a0e5991bbada78', 'sha256:ac9bb4c5ce3975aeac288cfcb5061ce60e0d14d92209e780c93954076c7c4367', 'sha256:de119f56f3c5f0e2fb4dee508531a32b069a5f2c6e827b272d1e0ff5ac040333', 'sha256:9b3152f2f5677b997ae6c804b73da05a39daa6a9e85a512e0e6823d81cdad7cc', 'sha256:e65610c5792870d45d7b68c677681376fcf9cc1c289f23e8e8b39c1485384185', 'sha256:3da8a678ca8b96c8606bbb8bfacd99a12ad5dd288bc6f7979baddd62f71c63ef', 'sha256:56ff08ab5df8429901ebdc5d15941b59f6253393cb5da07b4170beefcf1b2528', 'sha256:2a96c19c52ff442a808c105901d0bdfd2e28575b3d5f82e2f5fd67e20dc5f4ea', 'sha256:8ec53a0ea2a80c5cd1ab397925f94bff59222aa3cf9c6da938ce05c9ec20428d', 'sha256:be6b3fdec5c62f2a67cb3f8c6dbf56bbf3f61c0f046f84645cd1ca73532ea051', 'sha256:e9fdc7ac0d42bc3ea78818557fab03af6181e076a2944f43c38684b4b6bed8e3', 'sha256:13414591ff516e04fcdee8dc051c13fd3db13b673c7a4cb1350e6b2ad9639ad3', 'sha256:b03917871bf859a81ccb180c9a2e6c1e04d2f6a51d953e6a5cdd70c93d4e5a2a', 'sha256:75df5ef94c3fdc393c6b19d80e6ef1ecc9ae2f4263c09cacb178d871c02a5ba9'}), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=yarl, path=None, uri=None, extras=(), markers=python_version >= '3.7', vcs=None, specifier===1.9.2, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=yarl==1.9.2)>, _ireq=None)]
 Package installation failed...

Important! If I remove the second source from the Pipfile then pipenv installs successfully.

---

$ pipenv --support

Pipenv version: <module 'pipenv.__version__' from '/home/rl/.local/lib/python3.8/site-packages/pipenv/__version__.py'>

Pipenv location: '/home/rl/.local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/bin/python3'

OS Name: 'posix'

User pip version: '23.1.2'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.10',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.15.0-73-generic',
 'platform_system': 'Linux',
 'platform_version': '#80~20.04.1-Ubuntu SMP Wed May 17 14:58:14 UTC 2023',
 'python_full_version': '3.8.10',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• SESSION_MANAGER
• QT_ACCESSIBILITY
• COLORTERM
• XDG_CONFIG_DIRS
• XDG_MENU_PREFIX
• GNOME_DESKTOP_SESSION_ID
• MANDATORY_PATH
• LC_ADDRESS
• GNOME_SHELL_SESSION_MODE
• LC_NAME
• SSH_AUTH_SOCK
• XMODIFIERS
• DESKTOP_SESSION
• LC_MONETARY
• SSH_AGENT_PID
• GTK_MODULES
• GUAKE_TAB_UUID
• PWD
• LOGNAME
• XDG_SESSION_DESKTOP
• XDG_SESSION_TYPE
• GPG_AGENT_INFO
• XAUTHORITY
• DESKTOP_STARTUP_ID
• GJS_DEBUG_TOPICS
• WINDOWPATH
• HOME
• USERNAME
• IM_CONFIG_PHASE
• LC_PAPER
• LANG
• LS_COLORS
• XDG_CURRENT_DESKTOP
• VTE_VERSION
• INVOCATION_ID
• MANAGERPID
• GJS_DEBUG_OUTPUT
• LESSCLOSE
• XDG_SESSION_CLASS
• TERM
• LC_IDENTIFICATION
• DEFAULTS_PATH
• LESSOPEN
• USER
• DISPLAY
• SHLVL
• LC_TELEPHONE
• QT_IM_MODULE
• LC_MEASUREMENT
• XDG_RUNTIME_DIR
• LC_TIME
• JOURNAL_STREAM
• XDG_DATA_DIRS
• PATH
• GDMSESSION
• DBUS_SESSION_BUS_ADDRESS
• GIO_LAUNCHED_DESKTOP_FILE_PID
• GIO_LAUNCHED_DESKTOP_FILE
• LC_NUMERIC
• OLDPWD
• _
• PYTHONDONTWRITEBYTECODE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/rl/.local/bin/:/home/rl/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/rl/Temp/install

---

Contents of Pipfile ('/home/rl/Temp/install/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu117"
verify_ssl = true
name = "downloadpytorch"

[packages]
"pyannote.audio" = "*"

[dev-packages]

[requires]
python_version = "3.10"

Contents of Pipfile.lock ('/home/rl/Temp/install/Pipfile.lock'):
Pipfile.lock.txt

[matteius]

You need to add secondary sources explicitly to the Pipfile -- only the first source is tried for packages that do not specify an index explicitly per index restricted packages.

Try:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu117"
verify_ssl = true
name = "downloadpytorch"

[packages]
"pyannote.audio" = "*"
torch = {version="*", index="downloadpytorch"}

[dev-packages]

[requires]
python_version = "3.10"

[RomanSteinberg]

@matteius sure! But I demonstrate the problem when no secondary source is used.

[RomanSteinberg]

@matteius just checked your Pipfile. Installation fails either.

[matteius]

@RomanSteinberg

I reproduced what you are saying, seems odd -- I am noting that when you have the extra index in this example, something is pulling page from the second index even though it wasn't specified in Pipfile:

LinkCandidate('https://download.pytorch.org/whl/cu117/torchaudio-0.13.1%2Bcu117-cp310-cp310-win_amd64.w
hl (from https://download.pytorch.org/whl/cu117/torchaudio/)'))

This is causing it to also pull torch from pytorch.org but the name is different there -- you get:

<             "version": "==0.13.1+cu117"
---
>             "version": "==0.13.1"

Its actually failing to generate the hashes because its looking for hashes under the key 0.13.1 but they are actually present under 0.13.1+cu117 -- I am not sure why yet its even searching there but the package name difference kind of makes sense given you are including the cu117 wheel index, but yes I cannot explain why it would even by finding anything there without specifying it.

[RomanSteinberg]

@matteius did you tried Pipfile with torch package line (your version) or not (my version)? Anyway I think it is more important to solve problem in my state because the second source should have no influence on the install process.

[matteius]

I believe what happened is at some point pip changed behaviors from always preferring pypi to searching (randomly?) the provided sources. So the problem is really in my index restricted packages implementation, but there seems to be a simple fix within the pip patch I provide for index restricted packages -- there should be an else conditional that when a package index is not specified it will only search the first index, which should be the default index in the Pipfile. Making this small change to our patch fixes both example usages (mine and yours).

[matteius]

In fixing this, and seeing the test failure, I realize there is a larger implication: --skip-lock doesn't work with multiple indexes, and in general should be deprecated for removal. It encourages users to bypass the lock resolution which provides security and information that is critical for batch_install to know which index to supply for sub-dependencies (which is determined through the resolve phase).

Additionally, it seems install_search_all_source = true turns problematic when the resolved hashes are different from where they are being installed from. In order to get the installer to work properly with the other failing test I had to set install_search_all_source = false which is equivalent to removing that directive (default behavior). I imagine there will be some questions about this, but I feel strongly that fixing this bug in index restricted packages along with deprecating --skip-lock is the right direction to head.

[RomanSteinberg]

@matteius I would like to comment your last message. I had a case when it was impossible to install packages without --skip-lock and possible with it. It is hard for me to reproduce it now and provide you a minimal example. But, please, do not hurry with this flag deprecation.

[matteius]

@RomanSteinberg Well it should still be possible to install specific packages with pipenv run pip install package in the future to trouble shoot adding a specific package, along with the pipenv graph command. For now the --skip-lock flag is still present, just deprecated.