Random coverage failures

[zeux]

We keep seeing occasional errors during coverage runs on https://github.com/Roblox/luau project.

Here's the latest:
https://github.com/Roblox/luau/runs/4299224527 (11/23)

Here's more:
https://github.com/Roblox/luau/actions/runs/1490911436 (11/22)
https://github.com/Roblox/luau/actions/runs/1488639950 (11/21)
https://github.com/Roblox/luau/actions/runs/1481803053 (11/19)

This happens on different PRs and doesn't seem to happen on master builds.

The action setup (which simply uses the GHA provided by Coveralls):
https://github.com/Roblox/luau/blob/master/.github/workflows/build.yml#L77-L81

In all cases the error log looks like this (with different hashes):

Using lcov file: ./coverage.info
Error:  "2021-11-22T04:22:45.062Z"  'error from getOptions'
Error: Error: Command failed: git cat-file -p 8089ae44597a4c95ac193ce0cc2efc28c8036649
fatal: Not a valid object name 8089ae44597a4c95ac193ce0cc2efc28c8036649

I initially thought this is a one-off but this happens with alarming repeatability in the last week.

[afinetooth]

Hi @zeux.

We've seen this occasionally, but aren't sure of the cause because it's intermittent.

The line that throws the error appears to be this one.

It indicates the local git repo has no ref as given. Similar to what's described here when the ref is a branch name, like "master."

The best next step will be to re-run those builds (or run some new ones) in verbose mode, which will add debug info to your build logs, which should present some clues.

The way to invoke verbose mode for the Coveralls Github Action is to add the following CI environment variable to your GHA config yaml so that it's available to your coveralls step:
NODE_COVERALLS_DEBUG=1

The "NODE_" part is there because the Coveralls Github Action uses the node-coveralls integration under-the-hood, which is also where that line of code comes from.

Please share the verbose build log the next time this happens and we'll look deeper.

[afinetooth]

@zeux on the hunch your CI is (occasionally?) building in detached HEAD mode (which would explain why the ref doesn't exist at Github---because CI is creating an ephemeral commit just for the current build), it could be worth trying the suggestions here or here (to "recover" from detached HEAD state) to see if that fixes it.

Again, your verbose build log(s) will help uncover whether that's the issue.

[zeux]

Here's a new instance of this which comes with verbose logs that we've enabled:

https://github.com/Roblox/luau/runs/4435306714?check_suite_focus=true

Note that as per above we're simply using GitHub Actions, along with checkout@v1 without arguments; if something is creating a detached HEAD it's the action itself but that would seem surprising.

Also attaching zipped logs from the CI run just in case.

coverage.zip

[zeux]

The logs don't seem to show anything beyond the error; maybe it would make sense to print GHA-supplied hashes as well as a git log for the last 10 commits? I'm not sure where the hash noted in this run is even coming from.

It's puzzling to me that this issue is spurious. It feels like checkout action either detaches the head or doesn't, and in either case we'd either see the problem always or never...

[afinetooth]

@zeux you're right, it does seem like the issue would be persistent if that's the case.

I found this (solved) issue on the github community board, which mentions intermittently getting incorrect values for GITHUB_SHA when using [email protected]:

• GITHUB_SHA not always correct.

The solution recommends upgrading to version >= 2.2.0 of the checkout action.

The Coveralls Github Action uses GITHUB_SHA if you don't pass a value for its git-commit parameter. Here's the line where it picks that up. And here's where it would be replaced if you passed a value for git-commit.

FWIW, the Coveralls Github Action hands over its inputs for processing to the node-coveralls integration (which as I mentioned above, is running under-the-hood), and here is where the error you're seeing gets thrown by that code.

This is the best I have at the moment. Figured it's worth trying.

[afinetooth]

Here's more background on the same issue from Coveralls Github Action issue board:
coverallsapp/github-action#55

[zeux]

Thanks! This sounds very similar. I'm going to assume that this is checkout@v1 related and speculatively close this. We've switched to checkout@v2 and added a few more debug logs to the build step in question - if it does fail again I'll reopen this with more information (which would likely be 2022 at this point).

[afinetooth]

@zeux sounds good. Good luck.