Downstream tests download to wrong subdir (breaks boa+libsolv>=0.7.23)

[mbargull]

What happened?

See conda/conda-index#89 and linked issues.

This comes from openSUSE/libsolv@cc2da2e#diff-2f0af950e6d2249e0340147b80f5b8194de5362f23026ff65cdb65dfa57696aaR314-R320 , i.e., a newly added strict check in libsolv=0.7.23 update.
That check is probably reasonable/good -- but conda-build is not so strict and dumps the to-be-downstream-tested packages in the current subdir.

In conda-forge/conda-feedstock#197 (comment) the noarch package conda-smithy gets downloaded to conda-build/linux-64 which is then indexed and thus results in a repodata.json for linux-64 with a noarch package.

---

I didn't dig deep enough, but I guess somewhere around
https://github.com/conda/conda-build/blob/3.23.3/conda_build/build.py#L3144
-> https://github.com/conda/conda-build/blob/3.23.3/conda_build/render.py#L311
is the culprit.

[mbargull]

Until this bug is fixed, conda-forge/boa-feedstock#61 should act as a workaround.

[dholth]

conda-build intentionally combines noarch and native arch packages in an index here

conda-build/conda_build/index.py

Lines 164 to 165 in bd49d73

	# replace noarch with native subdir - this ends up building an index with both the
	# native content and the noarch content.

[kenodegard]

It looks like this is unrelated to libsolv but is rather some other change in mamba that started to check whether the package's subdir matches that of the repodata.json's subdir (I haven't pinpointed the exact change).

Conda-build rearchives/copies the downstream test's package from the package chache into the $CONDA_PREFIX/conda-bld index and uses the host subdir. I believe it happens in this hack (see line 220):

conda-build/conda_build/render.py

Lines 202 to 223 in 44defed

	def find_pkg_dir_or_file_in_pkgs_dirs(pkg_dist, m, files_only=False):
	_pkgs_dirs = pkgs_dirs + list(m.config.bldpkgs_dirs)
	pkg_loc = None
	for pkgs_dir in _pkgs_dirs:
	pkg_dir = os.path.join(pkgs_dir, pkg_dist)
	pkg_file = os.path.join(pkgs_dir, pkg_dist + CONDA_PACKAGE_EXTENSION_V1)
	if not files_only and os.path.isdir(pkg_dir):
	pkg_loc = pkg_dir
	break
	elif os.path.isfile(pkg_file):
	pkg_loc = pkg_file
	break
	elif files_only and os.path.isdir(pkg_dir):
	pkg_loc = pkg_file
	# create the tarball on demand. This is so that testing on archives works.
	with tarfile.open(pkg_file, 'w:bz2') as archive:
	for entry in os.listdir(pkg_dir):
	archive.add(os.path.join(pkg_dir, entry), arcname=entry)
	pkg_subdir = os.path.join(m.config.croot, m.config.host_subdir)
	pkg_loc = os.path.join(pkg_subdir, os.path.basename(pkg_file))
	shutil.move(pkg_file, pkg_loc)
	return pkg_loc

See #4763 for a potential fix

[kenodegard]

I now understand that the subdir checks on repodata.json was added to libsolv (not mamba). Either way, the investigations done above and the potential fix is valid.

[jezdez]

@kenodegard Should we close this here, or move the ticket to the conda/conda repo?

[jezdez]

Closing as I think this was fixed in #4763

[kenodegard]

Looks like we missed a second spot where the subdir is incorrectly set, this causes downstream tests to fail:

conda-build/conda_build/build.py

Lines 2458 to 2459 in 424b736

	if package_data['subdir'] != 'noarch':
	config.host_subdir = package_data['subdir']