Adding ability to install pip dependecies using private repo passed through index-url

[DevshreeG]

Hi
I would like to enhance pip support by enabling feature to add index-url and extra-index-url to environment.yml file under pip section. This would install the pip dependencies based on private repository or the extra repository.
This is similar to 135 but here I am suggesting to add the url's to the environment.yml file itself rather than in poetry.toml file.

For example:

# /tmp/environment.yml
channels:
  - conda-forge

dependencies:
  - pip
  - pip:
    - --index-url https://(private-repo-url)
    - --extra-index-url https://pypi.python.org/simple
    - notebook>=6.4.4,<7.0.0`

I would also like to contribute to this if this issue is accepted and would like some guidance on the same.

[DevshreeG]

Looks like if i add remote repository with URL passed from index-url and add it to the pool at here , then the pypi_solver should be able to find packages in the private repo.

Something like this

pypi = PyPiRepository()
remote = RemoteRepository("https://privaterepo.com/simple/")
pool = Pool(repositories=[remote, pypi])

[mariusvniekerk]

That looks like it should work. Please open a PR to add support for this.

[DevshreeG]

@mariusvniekerk I am trying to run the below by passing the private repo URL as we discussed above
Command : conda-lock -f conda_lock_test_config.yml -p linux-64
config.yml :

channels:
- conda-forge
dependencies:
- python==3.9.7
- pip>=21.1.3,<22.0.0
- pytest>=6.2.4,<6.3.0
- scikit-learn>=1.0.1,<1.1.0
- xgboost>=1.5.1,<1.6.0
- gcsfs==2021.11.0
- dask>=2021.10.0,<2021.11.0
- dask-ml>=2021.10.17,<2021.11.0
- shap>=0.40.0,<0.41.0
- matplotlib>=3.4.3,<3.5.0
- seaborn>=0.11.2,<0.12.0
- hyperopt>=0.2.5,<0.3.0
- freezegun>=1.1.0,<1.2.0
- grpcio==1.43.0
- protobuf>=3.19.4,<4.0.0
- datadog>=0.44.0,<0.45.0
- pip:
  - shopify-package1[trino, pipelines]==0.9.0
  - shopify_package>=1.1.2,<1.2.0
  - ray[default,tune]>=1.9.0,<1.10.0
  - tune-sklearn>=0.4.1,<0.5.0
  - jupyterhub>=1.4.2,<1.5
  - jupyterlab>=3.1.14,<4.0.0
  - notebook>=6.4.4,<7.0.0
  - duckdb==0.3.0
  - aiohttp>=3.7.0,<3.8.0
  - pandas>=1.4.0,<1.5.0
  - pyarrow>=6.0.0,<7.0.0
  - prometheus_client<0.14.0

I am getting below error
poetry.mixology.failure.SolveFailure: Because -dummy-package- depends on shopify_package (<2.0.0,>=1.0.0) which doesn't match any versions, version solving failed.

Any advice?

[shopigarner]

Just created a PR that should solve this:
#201

[ctcjab]

I see that #201, which the previous comment said should solve this, was merged 2 years ago. But the SolveFailure ("-dummy-package- depends on") error reported above still reproduces with the latest stable release (2.5.7).

Looking closer at #201, it does not address configuring different index urls via environment.yml (which is the feature that this issue is tracking), only via global ~/.config/pypoetry config.

Given that #201 has been implemented, would the remaining logic to get this to work be easy to add?

[maresb]

Hey @ctcjab, have you tried using pip-repositories as described here?

For working with mixed pip/conda dependencies, you may find pixi more stable. See also #708.