Bug Report: Successful codecovcli upload-process failing to process by codecov which introduces unnecessary friction and lack of observability

[chriscerk]

Describe the bug
A clear and concise description of what the bug is.

Environment

• Browser: n/a
• System: codecov cli from CI (uploading coverage reports)

To Reproduce
Steps to reproduce the behavior:

1. Run codecovcli --verbose upload-process

Your upload is now processing.
When finished, results will be available at: OUR_ORG_LINK_TO_RESULTS
Process Upload complete

3. Wait awhile...
4. See "Currently no uploads" in the UI and "Missing Head Report"

Expected behavior
A clear and concise description of what you expected to happen.

Uploading code coverage to work.

⭐ If you have metrics on:

• queued
• total processed
  • successfully processed
  • failed to process

That would be interesting to review since we see "Currently no uploads" which according to the codecov cli should not be true.

Showing the status in the UI would be great, eg if it is queued vs. processing vs. completed

---

update: from comment below

Also if code coverage doesn't process within a reasonable timeframe - it would much less disruptive for our teams to have a codecov yaml config setting to handle these processing delays - similar to informational: true.

For example, informational_on_processing_delay: true.

This is in addition to accurately tracking the status as mentioned above a few times.

---

Screenshots

n/a

Additional context

This has been working fine for awhile, but since the past week this has been extremely flaky and blocking many engineering teams.

[drazisil-codecov]

Hi @chriscerk

Can you please share the full SHA?

[chriscerk]

@drazisil-codecov 0731ef732d1df7971ba12832cb27598c80434c0e is one example

[drazisil-codecov]

@chriscerk can you open a ticket and share the full CLI output from your CI? Preferably with verbose enabled? Something is not happening correctly (don't think it's anything you are doing wrong), I but I don't know what...yet.

You can mention this thread, if you want. I'll get it :)

[davidlj95]

Same here, I can provide some examples as they're public repositories

I use Codecov GitHub Action to upload the reports there. And then have some PR checks that are mandatory to merge to the main branch. Recently (last week) I had to skip those checks given they were stuck in pending state. Investigating a bit, seems despite the action logs to upload the coverage report succeeded, no uploads appeared for that commit in Codecov's app. Hence Codecov didn't notify the results to GitHub and the check was left as pending.

An example from today:

• GitHub Action code coverage upload logs
• Codecov app reports no uploads for that commit

An example from another repository, 2 days ago:

• GitHub Action code coverage upload logs
• Codecov app reports no uploads for that commit

In both cases I forced a CI/CD run again by ammending the commit to change the date of the commit + force-pushing and it worked.

[drazisil-codecov]

@adrian-codecov Here are a few examples of uploads failing because create-commit didn't complete in time. Happy to huddle, I don't have any ideas for solutions at this time. Should I open an investigation ticket, or just use this one?

[drazisil-codecov]

Thanks, @davidlj95 ! Very helpful to be able to confirm that the CLI is calling all the steps in order. Definitely a server-side race condition. :-/

[drazisil-codecov]

#485 is related

[davidlj95]

Today happened again. Here are the details:

An example from today:

• GitHub Action code coverage upload logs
• Codecov app reports no uploads for that commit

Let me know if I'm spamming and I won't add moar comments 😛 Just providing info to confirm the issue stil exists + provide details to diagnose the issue

[chriscerk]

Just bumping this thread so it hopefully gets attention.

and to re-iterate from the ticket:

⭐ If you have metrics on:

• queued
• total processed
• successfully processed
• failed to process

That would be interesting to review since we see "Currently no uploads" which according to the codecov cli should not be true.

Showing the status in the UI would be great, eg if it is queued vs. processing vs. completed

We've had a lot of frustration from our teams and I don't have the time to track down every single instance of this.

[drazisil-codecov]

@souredoutlook , can we get some eyes on this, since I'm no longer in the support department?

[chriscerk]

@drazisil-codecov / @souredoutlook we're seeing more issues with codecov processing today again and it has been extremely disruptive.

We'd appreciate some eyes on this.

One example of many:

info - 2025-01-16 18:23:40,873 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/ConsultingMD/mx-virtual-visits/commit/fac65e6df4a22936cf3d7c4cab64024094126191
info - 2025-01-16 18:23:41,168 -- Process Upload complete

yet

[chriscerk]

Also if code coverage doesn't process within a reasonable timeframe - it would much less disruptive for our teams to have a codecov yaml config setting to handle these processing delays - similar to informational: true.

For example, informational_on_processing_delay: true.

This is in addition to accurately tracking the status as mentioned above a few times.

[chriscerk]

This is happening again...

both yesterday and today.

Please review the above recommendations so that internal incidents with codecov don't create unnecessary friction.

[rohan-at-sentry]

@chriscerk - I'm the PM on Codecov.

We're working on ways to improve the commits page to show processing status of individual reports - and surface errors more visibly on the PR comment and UI. You can follow along, and leave feedback here -- codecov/engineering-team#3229

That said, the point about informational_on_processing_delay is great. Let me take this back to the team !