TensorFlow job in Bazel CI is red but only in Ubuntu 14.04

[lfpino]

The TensorFlow job is failing with:

ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/817efe17b83eb78b343080fc9971c47d/external/local_config_python/BUILD:166:1: Couldn't build file external/local_config_python/numpy_include/numpy/__multiarray_api.h: Executing genrule @local_config_python//:numpy_include failed: Output bazel-out/k8-opt/genfiles/external/local_config_python/numpy_include/numpy/__multiarray_api.h is a symbolic link. Only regular files and directories may be uploaded to a remote cache. Change the file type or add the "no-cache" tag/execution requirement.

Looks like a remote execution issue, assigning to @buchgr to diagnose.

Full log in: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/179#415760a7-1156-4b75-897e-c2ed29db585a

[buchgr]

This is due to a recent change to remote caching, where we no longer can cache actions that generate symlinks as outputs. I ll take a look.

[buchgr]

cc @meteorcloudy

[meteorcloudy]

Since it's only failing since Wednesday, it should be caused by some change between (b8765a6, 73088a8]

[meteorcloudy]

This is a regression, so it will be a release blocker for 0.13.0, but not for 0.12.0 because it's cut before the breaking change.

[lfpino]

TF is back to green today (https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/180) but we still need to investigate what's going on here.

[vladmos]

There's a similar issue with rules_typescript: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/188#6b1ff1cb-296d-41ad-9743-2346d618001f

[jin]

TF job is red for both 14.04 and 16.04 now, with a different error.

ERROR: /var/lib/buildkite-agent/builds/buildkite-ubuntu1604-xddx-1/bazel-downstream-projects/tensorflow/tensorflow/tools/api/generator/BUILD:27:1: Couldn't build file tensorflow/tools/api/generator/api/__init__.py: Executing genrule //tensorflow/tools/api/generator:python_api_gen failed (Exit 1)
--
  | Traceback (most recent call last):
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/6b4c5d5813aa5928a7e6e5dae494f9c5/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/tools/api/generator/create_python_api.py", line 26, in <module>
  | from tensorflow.python.util import tf_decorator
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/6b4c5d5813aa5928a7e6e5dae494f9c5/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 63, in <module>
  | from tensorflow.python.framework.framework_lib import *  # pylint: disable=redefined-builtin
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/6b4c5d5813aa5928a7e6e5dae494f9c5/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/framework/framework_lib.py", line 104, in <module>
  | from tensorflow.python.framework.importer import import_graph_def
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/6b4c5d5813aa5928a7e6e5dae494f9c5/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/framework/importer.py", line 32, in <module>
  | from tensorflow.python.framework import function
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/6b4c5d5813aa5928a7e6e5dae494f9c5/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/framework/function.py", line 37, in <module>
  | from tensorflow.python.ops import variable_scope as vs
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/6b4c5d5813aa5928a7e6e5dae494f9c5/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/tools/api/generator/create_python_api.runfiles/org_tensorflow/tensorflow/python/ops/variable_scope.py", line 24, in <module>
  | import enum  # pylint: disable=g-bad-import-order
  | ImportError: No module named enum

14.04: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/197#32044fa3-fa5f-49ac-b157-6e59b8e8df04
16.04: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/197#7aaa1cc0-0dd9-4a42-baf1-6479e8cc93d4

[jin]

TF_serving has updated their TF commit hash, but now they're running into the no-cache issue:

ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/83a97a49514b3c6bb07710cea33f38f9/external/local_config_python/BUILD:166:1: Couldn't build file external/local_config_python/numpy_include/numpy/__multiarray_api.h: Executing genrule @local_config_python//:numpy_include failed: Output bazel-out/k8-fastbuild/genfiles/external/local_config_python/numpy_include/numpy/__multiarray_api.h is a symbolic link. Only regular files and directories may be uploaded to a remote cache. Change the file type or add the "no-cache" tag/execution requirement.
--
  | ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/83a97a49514b3c6bb07710cea33f38f9/external/local_config_python/BUILD:39:1: Couldn't build file external/local_config_python/python_include/Python-ast.h: Executing genrule @local_config_python//:python_include failed: Output bazel-out/k8-fastbuild/genfiles/external/local_config_python/python_include/Python-ast.h is a symbolic link. Only regular files and directories may be uploaded to a remote cache. Change the file type or add the "no-cache" tag/execution requirement.

14.04: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/197#5a04d42a-f730-4670-9852-7e763d9ef12e
16.04: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/197#5c639bd6-1df7-4f42-816b-638a208f0d41

[meteorcloudy]

Ping @buchgr

[meteorcloudy]

Any progress on this? @buchgr
Are we making the "no cache" issue for symlink a warning instead of an error?

[buchgr]

I have rolled back fa36d2f48965b127e8fd397348d16e991135bfb6 yesterday. Can you cherry pick this commit into the release?

[meteorcloudy]

Thanks, I'll cherry-pick it!

[arthur309]

@jin Is there any way to solve the build problem of "python_api_gen failed" on 14.04?

[jin]

@arthur309 sorry, I don't know what you're referring to. Is that error related to the remote execution issue in this thread?

[meteorcloudy]

This issue has been resolved by fa36d2f