CI (Buildkite): enable the commit status for asan

[DilumAluthge]

After #42229, the asan job should be passing reliably. Let's enable the commit status, so that people can e.g. see the status on PRs and master.

The asan job runs in the following situations:

1. Pushes to master
2. PRs to master

However, because the asan job is still in the "experimental" category, it is important to note that it will NOT run in the following situations:

1. Pushes to release-*
2. PRs to release-*

[KristofferC]

I kind of think that all these: "embedding", "llvmpasses", "analyzegc", "asan" etc should be one status check. And if one of them fails you go in and you check which one failed. There has been a bit of a status check inflation. Remember, we used to have two build checks: Windows (AV) and Travis (Linux + Mac). Now there are 25...

[DilumAluthge]

We could remove all of the individual Buildkite commit status, and just have a single commit status for Buildkite, named buildkite/julia-master.

(I would still keep the whitespace check separate, just so we can keep having it as a "required status check" in the GitHub branch protection settings for the master branch.)

[DilumAluthge]

@staticfloat What are your thoughts?

[DilumAluthge]

We could remove all of the individual Buildkite commit status, and just have a single commit status for Buildkite, named buildkite/julia-master.

#42284

[staticfloat]

Can we combine a subset of jobs under a single status? E.g. just set the github_commit_status.context to "sanitizers" for multiple jobs? I'm not sure what will happen (e.g. if they will overwrite eachother, or by intelligently combined) but it could be a good solution if it works.

[DilumAluthge]

Can we combine a subset of jobs under a single status? E.g. just set the github_commit_status.context to "sanitizers" for multiple jobs? I'm not sure what will happen (e.g. if they will overwrite eachother, or by intelligently combined) but it could be a good solution if it works.

We can combine them by setting up "dummy jobs" that depend on each job in the relevant subset.

For example, I could make a job named analysis that depends on the following jobs:

• analyzegc
• asan
• embedding
• llvmpasses

The analysis job itself would not do anything, e.g. its command step would just be :. But it would have a job dependency on each of the above jobs. Then, the analysis commit status would only be green if all of those four dependencies passed.

What kinds of groups would we want? Here is one proposal. The top-level bullet-points (in bold) are the commit statuses that would actually show up on GitHub, while the sub-bullet points are the actual jobs (which don't have commit statuses).

• analyze
  • analyzegc
  • asan
  • embedding
  • llvmpasses
• docs
  • doctest
  • docdeploy
• package
  • package_linux32
  • package_linux64
  • package_win32
  • package_win64
  • package_macos64
  • ...
• test
  • tester_linux32
  • tester_linux64
  • tester_win32
  • tester_win64
  • tester_macos64
  • ...

[vchuravy]

embedding and llvmpasses are not analysis checks, I think we only wanted to run them on one platform which is why they are separated but they are part of the proper testsuite.

[DilumAluthge]

embedding and llvmpasses are not analysis checks, I think we only wanted to run them on one platform which is why they are separated but they are part of the proper testsuite.

Ah. So would you say to put them in the test group? Or would we make a separate group?

[vchuravy]

test group should be good.

[tkf]

I noticed a couple of timeout failures before #41606 (comment) during ASAN build. But looking at recent ASAN builds in https://buildkite.com/julialang/julia-master-experimental/builds?branch=master they are all green. Also, going through about 10 pages, all past builds seem to only take 1 hour. Currently, we have a 2-hour limit. So, enabling this sounds good to me.

[staticfloat]

We can combine them by setting up "dummy jobs" that depend on each job in the relevant subset.

This will be useful for the future as well, where we want to, for instance, deploy things only after all test jobs have been successful.

[DilumAluthge]

We just need to decide on the groups.

Does the above grouping sound good for a first pass?

[staticfloat]

Yeah, sounds good to me. I'm not sure if it's good or bad to not be able to see platform differences at a glance, but let's go with this for now.

[DilumAluthge]

Cool. I'll add those changes to #41707.

[vtjnash]

If we could be really clever, I would say we only post each stage that failed + "test" + "whitespace"

[DilumAluthge]

If we could be really clever, I would say we only post each stage that failed + "test" + "whitespace"

Hmmmm. I don't know if a way that we can post only the stages/groups that failed. But I think (but am not 100% sure) that I can configure it to post only the individual jobs that failed.

So, for example, suppose that the tester_win32 and tester_linux32 jobs failed, but all other jobs passed. I can configure it so that GitHub only shows the following commit statuses:

• buildkite/julia-master (this status summarizes all of the Buildkite jobs, so in this example, it will be ❌)
• tester_win32 (which will be ❌)
• tester_win64 (which will be ❌)
• whitespace (which will be ✅)

Now suppose instead that all jobs passed. GitHub will only show the following commit statuses:

• buildkite/julia-master (this status summarizes all of the Buildkite jobs, so in this example, it will be ✅)
• whitespace (which will be ✅)

@vtjnash @staticfloat What do you think?

[DilumAluthge]

(I'm not actually sure if this is possible. I'm going to test it out.)

[vtjnash]

seems good. though I guess it is partly worth noting that github already hides all status if they are all green

[DilumAluthge]

It's a moot point anyway. It turns out that Buildkite doesn't let you define GitHub commit statuses conditionally.

Take a look at #42287, which has:

steps:

  - label: "i_will_pass"
    commands: |
      exit 0
    timeout_in_minutes: 20
    notify:
      - github_commit_status:
          context: "i_will_pass"
        if: build.state != "passed"

  - label: "i_will_fail"
    notify:
      - github_commit_status:
          context: "i_will_fail"
        if: build.state != "passed"
    commands: |
      exit 1

Unfortunately, as you can see in the commit status section at the bottom of #42287, both the i_will_pass and i_will_fail commit status are shown, even though the i_will_pass commit status passed.

So I think we'll need to go with the original plan of having a small number of "groups" or "categories".