'Exec failed due to IOException' error on tree artifact containing symlink to directory when --remote_download_outputs=toplevel

[ljessendk]

Description of the bug:

When --remote_download_outputs=toplevel (default in Bazel 7) I get an IOException whenever I try to build a target that depends on a rule producing a tree artifact with a symlink to a directory when the outputs of the rule that produces the tree artifact is fetched from cache.

The error I get looks like this:
ERROR: /user/lsje/Lars/toplevel/project_a/BUILD:7:8: Executing genrule //:consume failed: Exec failed due to IOException: /user/lsje/.cache/bazel/_bazel_lsje/46fd854a93d2f0b0c14f3170daa62d94/execroot/_main/bazel-out/k8-fastbuild/bin/symlink_to_directory/symlink_to_dir (No such file or directory)

I can get the above error to disappear if I either:

• Set --remote_download_outputs=all
• Build without cache
• Build when cache is not populated

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

tree_artifact_with_folder_symlink.bzl:

def _tree_artifact_with_folder_symlink(ctx):

    args = ctx.actions.args()

    outdir = ctx.actions.declare_directory(ctx.attr.name)

    ctx.actions.run_shell(
        outputs = ([outdir]),
        use_default_shell_env = True,
        command = "\n".join([
            "mkdir -p {outdir}/dir",
            "touch {outdir}/dir/file",
            "ln -s dir {outdir}/symlink_to_dir",
        ]).format(outdir = outdir.path)
    )

    return [
        DefaultInfo(files = depset([outdir])),
    ]

tree_artifact_with_folder_symlink = rule(
    implementation = _tree_artifact_with_folder_symlink,
)

BUILD:

load(":tree_artifact_with_folder_symlink.bzl", "tree_artifact_with_folder_symlink")

tree_artifact_with_folder_symlink(
    name = "symlink_to_directory",
)

genrule(
    name = "consume",
    srcs = ["symlink_to_directory"],
    outs = ["dummy"],
    cmd = "touch $@",
)

.bazelrc:

build --disk_cache /tmp/cache

Steps to reproduce (with Bazel 7.0.2):

rm -rf /tmp/cache
bazel clean
bazel build :consume
bazel clean
bazel build :consume

Which operating system are you running Bazel on?

Fedora 37

What is the output of bazel info release?

release 7.0.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

#20415 seems to have a similar reproduction example except for the consuming rule.

Any other information, logs, or outputs that you want to share?

No response

[philsc]

Very different reproduction, but the error is identical: aspect-build/rules_js#1412.
Possibly same problem maybe?

Again, not sure what's causing this issue, but it causes flaky builds and means we have to revert the 7.0.2 upgrade back down to 6.2.

[tjgq]

I haven't forgotten about this, but my plate is too full at the moment. I suspect it might be a simple fix that can still be done in time for 7.1.0. I'll add it as a soft release blocker.

[iancha1992]

@bazel-io fork 7.1.0

[iancha1992]

A fix for this issue has been included in Bazel 7.1.0 RC1. Please test out the release candidate and report any issues as soon as possible. Thanks!

[matthewjh]

@iancha1992 possible to re-open this one? I see the same issue on 7.1.0.

[tjgq]

@matthewjh Can you provide a repro? There have been a few different issues with symptoms similar to this one. (For what it's worth, the repro provided by the initial post in this thread does not crash with 7.1.0.)