RFC: Do not package tests by default

[h-vetinari]

I was unaware until recently that conda-build will also package whatever is being used in (& copied into) the test section as part of the final binary.

In general, I consider it best-practice to run the full upstream test suite against the package builds (and I've been doing this as much as possible on the feedstocks I'm involved with), but it's really unfortunate that an otherwise really good practice will then lead to extra package bloat (e.g. conda-forge/pillow-feedstock#117, conda-forge/scipy-feedstock#160), as well as other potential issues (e.g. conda-forge/pillow-feedstock#113)

I know that one of the features of conda is that one can invoke the tests of a given package from the CLI, but is that really a common usecase for packages by conda-forge? Why not use the conda-build feature --no-copy-test-source-files by default in our builds? Or perhaps coupled with a flag to switch it back on for those few cases that really need it.

Thoughts?

[jakirkham]

Usually this content is pretty small. For cases where test data gets larger, some folks have used split packages to split out binary test data (please see mpl_sample_data for example).

It can be useful for reproducibility locally. For instance some of the packages that we currently lack hardware to test we are able to test this way (please see conda build --test). We may also use this functionality in future CI infrastructure.

That said, could see potential value in having a conda-forge.yml setting to control whether test data is packaged.