Poetry install of torch+cuda build variants not possible through a clearml agent

[hamishc]

Poetry doesn't yet support torch+cuda build variants (python-poetry/poetry#2613), and they've indicated that it may take a while to implement (python-poetry/poetry#4221).

The current workaround is to just run poetry run python -m pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html after running poetry install, but when running through a clearml agent this isn't possible (as far as I can tell).

Is there a workaround for this issue that's usable in clearml, or is it not currently possible to use poetry + torch/cuda build variants + clearml?

[jkhenning]

Hi @hamishc ,

If you're using the agent in docker mode, you can specify custom commands to execute as a bash script after the docker container has been started and python has been installed. Take a look here: https://github.com/allegroai/clearml-agent/blob/6c5087e425bcc9911c78751e2a6ae3e1c0640180/docs/clearml.conf#L145

[hamishc]

The workaround above doesn't work if run before poetry install because poetry cannot yet inherit system packages, so adding scripts after docker has started doesn't help the issue. What we need is a way to run custom commands post-poetry, but before task execution.

[jkhenning]

Hi @hamishc, what would be the command run in such an entrypoint?

[hamishc]

To fix this issue I'd just need to run the pip install on the poetry venv above: poetry run python -m pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

More generally, we might want to run anything that needs post-poetry execution - like poe (https://github.com/nat-n/poethepoet) with poetry run poe task

[jkhenning]

I think this was just introduced in poetry - it is already supported, see python-poetry/poetry#2613 and https://stackoverflow.com/a/70342258 🙂

[hamishc]

If you use the marker / url constraint, you get this error when you try to install the torchvision build variant:

  SolverProblemError

  Because torchvision (0.9.1+cu113) depends on torch (1.8.1)
   and librephotos depends on torch (1.8.1+cu113), torchvision is forbidden.
  So, because librephotos depends on torchvision (0.9.1+cu113), version solving failed.

[jkhenning]

Hi @hamishc,

After some digging around, see here - as this is an actual poetry bug/issue (which I think they are working on resolving) - I would hate to add another layer that might break it...
There is a nice workaround (that is essentially, running a script, only this one is actually built into poetry, which is probably better as it will help users install on their local environment as well) - see python-poetry/poetry#2613 (comment)