How to check for failed status checks when there are duplicate names?

[Nuru]

we have a rule like

      - "-check-failure~=.*matrix.*"

Unfortunately, this rule passes when we have 2 status checks named "matrix" and one succeeds and one fails. Is that the intended behavior? I thought that if one fails, check-failure would be true, even though the other succeeded.

• I believe this is a bug, please fix it.
• In any case, how do we create the rule we want, which is for the condition to be true only when there are no failures of tests with "matrix" in the name?

More details in a follow-up comment below

[jd]

I think the answer lies in this paragraph: https://docs.mergify.com/conditions/#validating-all-status-checks

TL;DR: this is not possible and dangerous because when a PR is created, there is no failure of tests with "matrix" in the name, so the PR would be merged without any test run at all.
You need to list all the test you want to be passing, that's the only safe thing to do.

[Nuru]

@jd I provided only this snippet to focus the question. I have another rule that requires a specific test to pass.

I tested the PR in question (now merged and closed) against the configuration using the Mergify config editor. The 16 checks are all "status": "completed". The check-failure rule still does not catch the failed test, presumably because there is a successful test with the same name.

The name of the actual test has been changed to "matrix" for privacy, but otherwise...

I have a PR that has run, among other checks, 2 separate checks with the name "matrix". One failed and one succeeded (it was the same test but on different parts of the PR). It is listed in the status checks twice as such. Using the Mergify config editor to test rules against the PR:

• "check-success=matrix" is true (checked in results)
• "check-failure=matrix" is false (unchecked in results)
• "#check-success=12" is true despite the GitHub Web UI only showing 11 successful check and stating that 11 out of 16 checks passed
• "#check-failure=0" is true
• "#check-neutral=0" is true
• "#check-pending=0" is true
• "#check-skipped=0" is true

The correct numbers are 11 passed and 5 failed (2 failed, 3 cancelled). My suspicion is that the results are all off because the test names appear more than once in the list. (Probably the 12th success was the Mergify delete of the branch.) For every non-successful test, there was another check in the same check suite with the same name that was a success.

I would accept the result of both of check-success=matrix and check-failure=matrix tests passing, since there is a successful and a failed test with that name in the list of tests, but I do not accept check-failure not being true (or #check-failure=0) when there is a failed test. That to me is a bug.

So I would like the bug to be fixed, and until then, I would welcome any ideas about a workaround.

[jd]

The check-failure rule still does not catch the failed test, presumably because there is a successful test with the same name.

This should not be the case. If you can share a PR URL with that behaviour (you can send this to [email protected]) we can take a look at it and find the root cause of what you're seeing.

The fact the number of test passed is 11 vs 12 is also suspicious, but the GitHub UI might be something off compared to what the GitHub API returns.

If you can share the PR URL that'd help understanding. :)

[Nuru]

I sent info requested via email.