gh-allow-mergeable-bypass-apply checks all statuses

[andrewjamesbrown]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

When running atlantis apply with --gh-allow-mergeable-bypass-apply enabled, Atlantis responds with "Apply Failed: Pull request must be mergeable before running apply.".

In the Github UI, all status checks (apart from atlantis/apply) are successful, but when querying the GH API, several statuses are not "state": "success";

e.g. (removed PII):

  {
    "url": "https://api.github.com/repos/org/myrepo/statuses/83fae9ddc727f1f34a81ebff1832cbdd0ae32f33",
    "avatar_url": "https://avatars.githubusercontent.com/u/xxxxx",
    "id": 123456,
    "node_id": "SC_kwDOHzjQYc8AAAAE-Sli9A",
    "state": "pending",
    "description": "in progress...",
    "target_url": "https://atlantis.mydomain/jobs/57ec77c0-0478-4748-ab0d-f85af4de26ff",
    "context": "atlantis/pre_workflow_hook: Pre workflow hook #0",
    "created_at": "2023-02-02T19:28:42Z",
    "updated_at": "2023-02-02T19:28:42Z",

Reproduction Steps

In my PR, first I tried to atlantis apply when someone other than a CODEOWNER approved the PR, and then got a codeowner to approve, and then atlantis apply again.

Note that we also have a GH Action that runs on every commit that formats/lints and re-commits back to the repo - possibly this may have caused these pending actions.

Logs

Environment details

Atlantis version: 0.22.3
Running Atlantis in Hashicorp Nomad

Additional Context

[andrewjamesbrown]

Looking at https://github.com/runatlantis/atlantis/pull/2436/files#diff-8f27c369fd18d57f78cad1008481a28ce841f08630c69d9c9bbec37c0574dd6aR323-R330, it looks like Atlantis iterates through all statuses (not just required ones for merge), and if any are !success it returns false. Since we have previous atlantis/plan that are pending for some reason in our statuses (and not showing for the latest commit in the UI), this check fails and Atlantis will not apply.

[jamengual]

I think we need to look at the problem of those statuses that stay as pending.
is this reproducible? can you actually make it fail in your prs?

Please feel free to propose a PR.

[stasostrovskyi]

I think that Repository Rules (changelog) will allow us to get rid of this whole feature. It should be possible to create a separate rule that requires atlantis/apply but allows Atlantis GH App to bypass it. This way we would be able to rely again on a single mergeable state of PR as presented by API.

A couple of considerations:

1. This will require Atlantis to use GitHub App (since GH doesn't have plans to support bypassing rules by specific users)
2. Repository Rules is in Beta and missing some crucial functionality (e.g bypass by GitHub Apps doesn't seem to work)
3. Since the use case (disallow users to merge tf PR until atlantis applied it) seems to be popular, documentation on how to setup such a rule would be needed.

[nitrocode]

Repository rules would be ideal.

@edbighead did some sleuthing and found that switching from branch protection to repository rules is not currently possible due to our reliance on GetBranchProtection

atlantis/server/events/vcs/github_client.go

Line 417 in 6ff0e2f

required, resp, err := g.client.Repositories.GetBranchProtection(context.Background(), repo.Owner, repo.Name, *pull.Base.Ref)

[stasostrovskyi]

It would be possible to scrape all this code behind the feature flag if only GitHub would implement GetMergeableState API that will take into account which user is calling it. This way we can add atlantis app or user to bypass certain ruleset(s) and let GitHub figure out whether a PR is mergeable for Atlantis or a developer.

Unfortunately, they know that they lack such API call but never promised to implement it :(

[usmonster]

It would be possible to scrape all this code behind the feature flag if only GitHub would implement GetMergeableState API that will take into account which user is calling it.

FYI, this is also what's blocking #4116 (see also this thread).