feat: only instantiate conda prefix for pypi solve on request

[nichmor]

Intro

Pixi has had PyPI dependency support for a while now, we do this by integrating with uv. As you might be aware, python has the concept of source dependencies and also for some older source distributions these might not contain static metadata. In general when static metadata is available, then a python interpreter is only needed at install time not at resolve time, however when projects use a setup.py an interpreter is needed during resolve, because the metadata needs to be built.

Additionally, projects like: https://github.com/facebookresearch/detectron2 even need certain libraries to be available (other than just python) during the solve. With pixi, one can make use of the instantiated conda prefix for this. This is one of the benefits when compared to a PyPI only solve, we can install the system dependencies we need. However, this is only needed in a select number of cases. What we did, however, is instantiate a conda prefix if that environment contained PyPI dependencies.

If your environments were big, this can take a significant amount of time.

What changed?

This PR changes that behavior, only when source distributions (i.e git, non-editable path, or sdist) is encountered, do we actually instantiate the prefix. If you do not have any source dependencies, we instantiate no prefix before doing a PyPI solve. You can further enforce this by adding no-build = true to your [pypi-options]. Also when a previously built sdist is cached, i.e. the built metadata is available, we also skip instantiation.

So we expect a much faster solve for tomls with little source dependencies and heavy environments. This verification can be seen in the comments. This will be the first step though, there are also repositories like rerun-io/rerun that actually do have numerous source dependencies but only require python for the most of them. A further PR could be introduced where we also only instantiate a partial prefix on solve, this does require some user-input, as we need to know which packages need to be available.

While this changes a lot of the pre-solve behavior, I (@tdejager) expect that not a lot should change, and that most cases should become faster. We could also enforce no-build = true in our CI (that have no source dependencies), so that we are sure we never built when do not need to. This again is out of scope for this PR.

How to test it?

You can use this pixi.toml. Running pixi install should install only default env, and py39 because it request sdist.

Do not forget to clean uv-cache! Otherwise, it is possible to skip entire installation of py39

[project]
authors = ["nichmor <[email protected]>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "partial-env"
platforms = ["osx-arm64"]
# platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]
python = "*"
rust = "*"
pytorch = "*"

[pypi-dependencies]
boltons = "*"
# pillow = { git = "https://github.com/python-pillow/Pillow.git" }
# sdist = "*"



[feature.py39.dependencies]
python = "3.9"

[feature.py39.pypi-dependencies]
boltons = "*"
sdist = "*"

[feature.py310.dependencies]
python = "3.10"

[feature.py310.pypi-dependencies]
boltons = "*"

[feature.py311.dependencies]
python = "3.11"

[feature.py311.pypi-dependencies]
boltons = "*"



[environments]
py39 = ["py39"]
py310 = ["py310"]
py311 = ["py311"]

[ruben-arts]

Just a quick test of this PR 😉

gh repo clone skops-dev/skops
cd skops

pixi g i pixi==0.40.3 -e pixi40_3 --expose pixi40_3=pixi

hyperfine -r 3 -w 1 -p 'pixi clean && rm -f pixi.lock ' 'pixi list' 'pixi40_3 list'

Results in:

Benchmark 1: pixi list
  Time (mean ± σ):     17.414 s ±  3.818 s    [User: 7.603 s, System: 15.594 s]
  Range (min … max):   14.126 s … 21.601 s    3 runs
 
Benchmark 2: pixi40_3 list
  Time (mean ± σ):     41.604 s ±  7.046 s    [User: 11.093 s, System: 297.530 s]
  Range (min … max):   33.746 s … 47.359 s    3 runs
 
Summary
  pixi list ran
    2.39 ± 0.66 times faster than pixi40_3 list

I think we can still make this way faster but we need to figure out why this still takes time. (its spend on building the editable)

[tdejager]

Because the build dispatch can now panic when getting the interpeter, we might need to catch this and re-throw to something nice. I guess we could just wrap that around the actual resolve call, ditto for the LookaheadSolver.

[tdejager]

@baszalmstra did an initial review myself and pointed out some improvements. Would be good if you can take a look.

Also note to self @tdejager, let's test this on rerun-io repo as well, in that case some prefixes should still be instantiated, cause its using a lot of source deps.

[tdejager]

Ok all seems good on https://github.com/rerun-io/rerun, pixi install succeeds and for a, heavily cached comparison (I ran install a couple of times), it takes:

pixi clean && rm pixi.lock && pixi install:
This PR: 16s
latest release: 32s
Timings are just the pixi install part.

So no performance regressions there. @jleibs this might be interesting for you guys as well :)

[ruben-arts]

Left some small remarks! Looking forward to this change! 🚀

[ruben-arts]

Please make an error type

[tdejager]

Sorry, will not be able to this in this PR as it would result in needing to refactor a lot of miette types in update. We need to this seperately. I can take the message from miette for update_prefix_conda and find_installed_packages and put them in an error but I do not believe this to be advantageous at this point.

[ruben-arts]

Isn't it a good idea to move this function here? Looks like it's in the wrong location now.

[tdejager]

I tried doing thsi refactor but this exploded into a major refactor, you've noticed a good point for sure. We should do that refactor afterwards. But I think this needs to be out of scope for this PR.

[ruben-arts]

I've continued testing, and found nothing wrong. I've got one question.

It's now faster even when you do have sdist as a dependency in my pypi-dependencies. Any idea why that would be?

[tdejager]

I've continued testing, and found nothing wrong. I've got one question.

It's now faster even when you do have sdist as a dependency in my pypi-dependencies. Any idea why that would be?

Could be that sdist is cached for that version and if that is the case, it does no instantation of the prefix during solve at all. Unsure what your test parameters were :)