fixing issue with bazel shutdown causing circleci hung

[huang195]

bazel shutdown is causing some clean up problems in circleci that's blocking CI.

[istio-testing]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: huang195
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: venilnoronha

If they are not already assigned, you can assign the PR to them by writing /assign @venilnoronha in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[huang195]

/hold

[huang195]

macos CI check is somewhat different from linux version, which uses a fixed 0.22 version of bazel. macos relies on brew which could install newer version of bazel. At the moment, it install 0.25 version of bazel.

Update: bazel doesn't support older versions via brew (https://github.com/bazelbuild/homebrew-tap). ugh

[huang195]

for macos, the flag --incompatible_bzl_disallow_load_after_statement=false is a wet bandaid that will eventually have to fall off. Its default is already to true. In the very near future, the flag will be removed, and all bzl files need to be modified to be compliant with this new check.

[huang195]

@PiotrSikora @lizan @qiwzhang can you help take a look at this PR? I'm looking to revert some of the changes from this PR (#1914) as it's causing CircleCI to consistently hang. I'm at a loss with macos as brew is installing bazel 0.25 and it's introducing quite a bit of incompatibility issues that older version of bazel 0.22 that we are using for linux does not have.

[huang195]

FYI, after digging around on github on recently merged PRs onto the master branch, I see that the last time macos CI ran successfully was when brew installed 0.23 version of bazel :(

[lizan]

add --incompatible_bzl_disallow_load_after_statement=false to mac bazel CLI should fix mac CI

[huang195]

@lizan yeah I tried to add the flag, but the build hit other problems, so I removed the flag. See this circleci build that had this flag set to false: https://circleci.com/gh/istio/proxy/6151:

ERROR: /Users/distiller/.cache/bazel/external/envoy/bazel/BUILD:113:1: error loading package '@com_github_grpc_grpc//src/proto/grpc/health/v1': in /Users/distiller/.cache/bazel/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl: Label '@com_github_grpc_grpc//:bazel/cc_grpc_library.bzl' crosses boundary of subpackage '@com_github_grpc_grpc//bazel' (perhaps you meant to put the colon here: '@com_github_grpc_grpc//bazel:cc_grpc_library.bzl'?) and referenced by '@envoy//bazel:grpc_health_proto'
ERROR: Analysis of target '//src/envoy:envoy' failed; build aborted: Analysis failed

Any idea what could be causing this problem?

[huang195]

/test proxy-presubmit-asan

[istio-testing]

@huang195: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
prow/proxy-presubmit-asan.sh	9f2ec21	link	/test proxy-presubmit-asan

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[huang195]

@lizan I feel like going down a rabbit hole chasing down all the dependencies that bazel 0.25 on macos is causing. I'm going to open another PR that fixes all the other CI cases except macos, and close this one. As it's not one of the "required" CIs, hopefully, it will at least not block all the PRs that would otherwise succeed.