Make check-manifest run safe in parallel with other processes running sdist

[philpep]

Hi,

We have a tox configuration including check-manifest and py3 environments that are run in parallel in our CI.
We have random failure (either on check-manifest or py3) due do sdist running in parallel in the same directory.

Would it be possible to use a temporary directory for most of check-manifest operations ?

Not sure how this can be handled in setuptools. tox itself has some code that allow running environment in parallel, it might worth checking how they do.

Thanks!

[mgedmin]

check-manifest runs setup.py sdist twice: once from the source directory, and once from a temporary directory (with a subset of the files copied over). It must run setup.py sdist from the source directory at least once, so it can discover files that get picked up by setup.py that are not versioned (see #27).

I'm surprised tox runs sdist at the same time it runs one of the environments. I always though tox ran the sdist once first, and then installed the built sdist into the various toxenvs in parallel.

I wonder if switching to pep517 builds would help with these kinds of problems, or if they are also not parallelizable?

I'm sorry, I'm not sure what to suggest for you.

(Also, apologies for a late reply, I was on vacation and was pretending computers didn't exist.)

[philpep]

Hi thanks for your reply !

Indeed, in theory tox run sdist once, one way to make the issue occur less frequently is to run "tox --sdistonly" and then run environments in parallel. But I think there's a possible concurrency issue that make tox re-run a sdist during the process (probably due to a file mtime/atime being modified).

Hopefully setuptools/distutils will allow building sdist in a parallel-safe manner in a nearly future.

Actually I'm not even sure the issue is related to check-manifest, so feel free to close the issue :)