Ensure local builds can use CI cache

[ajschmidt8]

The changes in #35 allow users to run CI test scripts locally by enabling the artifact transfer commands to work in a non-CI environment.

One thing that's still left to confirm is whether local builds are able to properly utilize the sccache that's been populated by CI.

This workflow is useful for the following scenario:

• A user opens a PR and the builds fail
• User wants to replicate the build failure locally while utilizing the build cache that was populated by CI

The changes from #35 should enable this to be possible, but I haven't yet been able to verify it locally.

Opening this issue as a reminder to confirm this.

[ajschmidt8]

One additional change that will need to be made across repositories for this to work is to add this SCCACHE_NO_CREDENTIALS environment variable to the conda recipes' script_env section (e.g. here).

We should also check that the inclusion of that environment variable doesn't interfere with CI's regular use of sccache.

My local testing suggests that, as long as that environment variable is not set in CI, it will not be passed through to the conda build environment and therefore should not interfere with regular sccache usage.

That environment variable is documented here: https://github.com/mozilla/sccache/blob/f8972dcc5570b9308847ea7c9b7c25e86e293fbf/docs/S3.md#L29.

[ajschmidt8]

This sccache PR should help with this effort:

• Make SCCACHE_S3_NO_CREDENTIALS require a value of true mozilla/sccache#1724

Once that PR is merged and released, we can add a ENV SCCACHE_S3_NO_CREDENTIALS=false line below:

• https://github.com/rapidsai/ci-imgs/blob/2d43b361db5a31dbaef3a5a68140ff811f9ba51d/Dockerfile#L17-L24

This way CI builds will not use the SCCACHE_S3_NO_CREDENTIALS feature, but local builds will use the feature, since it gets set to true for local builds here:

gha-tools/tools/rapids-env-update

Lines 13 to 19 in 01c94b2

	if [ "${CI:-false}" = "false" ]; then
	# Configure sccache for read-only mode since no credentials
	# are available in local builds.
	export SCCACHE_S3_NO_CREDENTIALS=true
	export PARALLEL_LEVEL=${PARALLEL_LEVEL:-$(nproc)}
	export RAPIDS_BUILD_TYPE=${RAPIDS_BUILD_TYPE:-"pull-request"}
	fi

).

Then we can add the SCCACHE_S3_NO_CREDENTIALS environment variable to the script_env section of all of our recipes:

• https://github.com/rapidsai/cudf/blob/d1cebdbf50d92358962bc66a1b2c0a8476de1f35/conda/recipes/libcudf/meta.yaml#L17-L29

After these changes, local builds should be able to seamlessly use the sccache cache that's populated by CI.

[ajschmidt8]

I also updated our rapids-sccache-east bucket to enable all objects to be publicly readable by default.

I've confirmed that enabling this setting allows local builds to use the rapids-sccache-east cache in combination with the SCCACHE_S3_NO_CREDENTIALS variable.

[ajschmidt8]

mozilla/sccache#1724 has been merged and released in https://github.com/mozilla/sccache/releases/tag/v0.4.2.

The next action items are:

• Ensure this new sccache version is incorporated in our CI images (for both conda and wheels)
  • conda - https://github.com/rapidsai/ci-imgs/actions/runs/4790811593
  • wheels - Update sccache version manylinux#3
• Add ENV SCCACHE_S3_NO_CREDENTIALS=false to our CI images here
• Add SCCACHE_S3_NO_CREDENTIALS to all of the relevant script_env sections of our recipes (e.g. here)

[ajschmidt8]

PRs:

• ci-imgs - Add SCCACHE_S3_NO_CREDENTIALS environment variable ci-imgs#51
• cudf - Enable sccache hits from local builds cudf#13248
• rmm - Enable sccache hits from local builds rmm#1257
• raft - Enable sccache hits from local builds raft#1478
• cugraph - Enable sccache hits from local builds cugraph#3526
• cucim - Enable sccache hits from local builds cucim#551
• cuspatial - Enable sccache hits from local builds cuspatial#1109
• ucxx - Enable sccache hits for local builds ucxx#42
• kvikio - Enable sccache hits for local builds kvikio#210

[ajschmidt8]

closing this issue now that all the PRs above are merged.