repeating downloads of scala library

[ittaiz]

Hi,
I have a problem where for some cases when I build some projects bazel downloads the scala library all over again.
The above description is vague since I'm not entirely sure on the circumstances and it very well might be an edge case of me working on many WORKSPACES with different bazel versions (release from brew and local dev).
Has anyone encountered this? Anything that can be done in this repo?
Sounds related to bazelbuild/bazel#1752 but not sure

[johnynek]

I do see it too. It is pretty frustrating.

It may also be related to: bazelbuild/bazel#2490

I have not tried the repository cache, but you can see reports of success here:
bazelbuild/bazel#1752 (comment)

[ittaiz]

I think, not sure, that this is related to test_run.sh
It seems that if I run bazel test after running it then bazel needs to kill the unresponsive server and then do some expensive stuff. Might be wrong. Don't know why yet

[softprops]

Is anyone looking at this? We're running into the same issue --experimental_repository_cache does look promising. I noticed the appengine rules recommended it.

[ittaiz]

@softprops What is your use-case? for me this happened a lot when I was switching between running test_run.sh and bazel test for rules_scala itself. I figured it was related to how test_run.sh is built but maybe this is a bigger issue.

[softprops]

My use case is running builds in ci. I'm caching the initial bazel installation in a container which is then used to run build scripts which call bazel. We're now caching build results with bazels remote cache suppprt but the last piece is to cache the loading of skylark rule defs and the artifacts they depend on. For each one of these ci builds, skylark rules are redownloaded along with the scala jar deps. I recently started investigating that and bumped into this thread.

[softprops]

Wanted to share a technique I've found helpful to reduce this download time and reduce chances of network errors connecting to github ( that has happened multiple times ) in our ci env.

We run bazel in a docker container whose image has already built a dummy workspace in a well known dir with scala rules and a few other deps.

When we run the container, we mount a checkout, for reasons, in a slightly different dir that's symlinked to the well known dir. Before we build we run the following which symlinks a cache dir to the one that's baked into the image. The net result is we no longer have to download rule defs or any of their dependencies on any of our ci builds. This has improved the reliability of our builds.

cd /well/known/dir

CI_WORKSPACE_HASH=$(echo -n "/well/known/dir" | md5sum | cut -d ' ' -f 1)
THIS_WORKSPACE_HASH=$(echo -n $(pwd -P) | md5sum | cut -d ' ' -f 1)
CACHE_DIR=${HOME}/.cache/bazel/_bazel_$(whoami)
ln -s ${CACHE_DIR}/$CI_WORKSPACE_HASH ${CACHE_DIR}/${THIS_WORKSPACE_HASH}

There may be a more straightforward way to do this but this has worked very well for us