PyPI dependencies raise error when falling back from osx-arm64 to osx-64

[ekiefl]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

This is the pixi.toml. It is a stripped down version of the "multiple machines from one project" example in the multi-environment docs

[project]
name = "my-ml-project"
description = "A project that does ML stuff"
authors = ["Your Name <[email protected]>"]
channels = ["conda-forge", "pytorch"]
# All platforms that are supported by the project as the features will take the intersection of the platforms defined there.
platforms = ["win-64", "linux-64", "osx-64"]

[dependencies]
python = "3.11.*"

[pypi-dependencies]
matplotlib-base = ">=3.8.2,<3.9"

[feature.cuda]
platforms = ["win-64", "linux-64"]
channels = ["nvidia", {channel = "pytorch", priority = -1}]
system-requirements = {cuda = "12.1"}

[feature.cuda.dependencies]
pytorch-cuda = {version = "12.1.*", channel = "pytorch"}

[feature.cpu]
platforms = ["win-64", "linux-64", "osx-64"]

[environments]
cuda = ["cuda"]
default = ["cpu"]

On a osx-arm64 machine, I get the following error:

▶▶ pixi install
 WARN osx-arm64 (Apple Silicon) is not supported by the pixi.toml, falling back to osx-64 (emulated with Rosetta)
  × Unable to solve pypi dependencies for the cuda environment because no compatible python interpreter can be installed for the current platform
   ╭─[7:13]
 6 │ # All platforms that are supported by the project as the features will take the intersection of the platforms defined there.
 7 │ platforms = ["win-64", "linux-64", "osx-64"]
   ·             ────────────────┬───────────────
   ·                             ╰── even though the projects does include support for 'osx-arm64'
 8 │
   ╰────
    ╭─[18:13]
 17 │ [feature.cuda]
 18 │ platforms = ["win-64", "linux-64"]
    ·             ───────────┬──────────
    ·                        ╰── feature 'cuda' does not support 'osx-arm64'
 19 │ channels = ["nvidia", {channel = "pytorch", priority = -1}]
    ╰────
  help: Try converting your [pypi-dependencies] to conda [dependencies]

If you move the matplotlib-base dependency into dependencies, it works.

Issue description

I believe this issue is related to #1404

Basically, on macOS, I have dependencies that don't have support for osx-arm64, which is what pixi determines my platform to be. So I have to fallback to osx-64 by dropping support for osx-arm64, which according to this warning is emulated via Rosetta:

 WARN osx-arm64 (Apple Silicon) is not supported by the pixi.toml, falling back to osx-64 (emulated with Rosetta)

This works well, except for cases like the MRE above. I think that this is a bug, because although the warning says that we're falling back to osx-64, the platform is still being treated as osx-arm64 during the pypi dependency resolution, which subsequently raises pixi errors.

Expected behavior

One should be able to add pypi dependencies when emulating osx-64.

[ruben-arts]

Hi @ekiefl

TL;DR The problem is the cuda environment. You can't solve [pypi-dependencies] for a windows/linux only environment on a mac.

This issue is actually a duplicate of: #1130 and #1051 in one.

The problem is not the apple silicon, that's just a warning you would get even with a working environment. Just to let you know pixi is going to use Rosetta.

The problem comes from pixi requiring a Python interpreter to solve it's [pypi-dependencies] it gets this from the [dependencies] which it has to install first. In your MRE you have a system-requirement on cuda and no osx-* platform defined. So pixi can not install the conda environment on your platform to solve the cuda environment.

To fix this we've got multiple ideas described in this bigger tracking issue: #1295.

But for the time being here are some work-arounds:

• Easiest: don't use [pypi-dependencies] at all, so we don't need to install an environment to solve.
• Keep the [pypi-dependencies] out of the cuda environment and split the cuda work from the normal work.
• Split the project into two folders: a cuda and non-cuda folder and give them separate pixi.tomls which can do their own solves on their respective systems.

I hope this helps, if so please close this issue in favor of the others.

[ekiefl]

Hey @ruben-arts,

Thanks for the clear explanation. Some of this is admittedly over my head, but I'm glad this is part of a known issue that's being worked on.

Also, thanks for suggesting these workarounds.

Closing.