Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: fix JSONDecodeError when combining .coverage.gw* files #4557

Merged
merged 1 commit into from
Jan 30, 2025
Merged

ci: fix JSONDecodeError when combining .coverage.gw* files #4557

merged 1 commit into from
Jan 30, 2025

Conversation

romanz
Copy link
Contributor

@romanz romanz commented Jan 30, 2025

For example: https://github.com/trezor/trezor-firmware/actions/runs/12976369614/job/36188913357

Run nix-shell --run "poetry run make -C core coverage"
make: Entering directory '/home/runner/work/trezor-firmware/trezor-firmware/core'
./tools/coverage-report
COVERAGE_THRESHOLD set to 77%
.coverage.core_click_test12
.coverage.core_click_test13
.coverage.core_click_test14
.coverage.core_click_test15
.coverage.core_click_test16
.coverage.core_click_test17
.coverage.core_device_test30
.coverage.core_device_test31
.coverage.core_device_test32
.coverage.core_device_test33
.coverage.core_device_test34
.coverage.core_device_test35
.coverage.core_fido2_test1
.coverage.core_monero_test2
.coverage.core_persistence_test2
.coverage.core_u2f_test2
.coverage.gw0
.coverage.gw1
.coverage.gw2
.coverage.gw3
Creating .coverage.empty file
Combining .coverage.* files
Coverage.py warning: Couldn't read data from '/home/runner/work/trezor-firmware/trezor-firmware/core/.coverage.gw0': JSONDecodeError: Extra data: line 1 column 79752 (char 79751)
Coverage.py warning: Couldn't read data from '/home/runner/work/trezor-firmware/trezor-firmware/core/.coverage.gw1': JSONDecodeError: Extra data: line 1 column 81746 (char 81745)
Coverage.py warning: Couldn't read data from '/home/runner/work/trezor-firmware/trezor-firmware/core/.coverage.gw3': JSONDecodeError: Extra data: line 1 column 78777 (char 78776)
ERROR: Code coverage is less than 77% - 74%
See core/htmlcov/index.html for details.
make: *** [Makefile:481: coverage] Error 1
make: Leaving directory '/home/runner/work/trezor-firmware/trezor-firmware/core'
Error: Process completed with exit code 2.

Single .coverage files were renamed, but make test_emu_ui_multicore generates multiple .coverage.gw* files, which may overwrite each other in case there are multiple similar CI jobs (e.g. when translation-related jobs are enabled).

@romanz romanz self-assigned this Jan 30, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 30, 2025 •
edited
Loading

core UI changes device test click test persistence test
T2T1 Model T test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
T3B1 Safe 3 test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
T3T1 Safe 5 test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
All main(screens)

@romanz
Copy link
Contributor Author

romanz commented Jan 30, 2025

Coverage upload & download seems to be working:

Device tests (T2T1, universal, noasan, en)

Run ./.github/actions/upload-coverage
Run for F in core/src/.coverage core/src/.coverage.gw*
mv: cannot stat 'core/src/.coverage': No such file or directory
renamed 'core/src/.coverage.gw0' -> 'core/src/.coverage.gw0.core_device_test0'
renamed 'core/src/.coverage.gw1' -> 'core/src/.coverage.gw1.core_device_test0'
renamed 'core/src/.coverage.gw2' -> 'core/src/.coverage.gw2.core_device_test0'
renamed 'core/src/.coverage.gw3' -> 'core/src/.coverage.gw3.core_device_test0'
Run actions/upload-artifact@v4
With the provided path, there will be 4 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
Uploaded bytes 91356
Finished uploading artifact content to blob storage!
SHA256 hash of uploaded artifact zip is 7b9bff2180d8eb386e48588311c7d99a66ad63f1a11fe33e5a1a1b9c00441e60
Finalizing artifact upload
Artifact core-coverage-T2T1-core_device_test-0.zip successfully finalized. Artifact ID 2509074054
Artifact core-coverage-T2T1-core_device_test-0 has been successfully uploaded! Final size is 91356 bytes. Artifact ID is 2509074054
Artifact download URL: https://github.com/trezor/trezor-firmware/actions/runs/13047700328/artifacts/2509074054

Device tests (T2T1, btconly, noasan, en)

Run ./.github/actions/upload-coverage
Run for F in core/src/.coverage core/src/.coverage.gw*
renamed 'core/src/.coverage' -> 'core/src/.coverage.core_device_test1'
mv: cannot stat 'core/src/.coverage.gw*': No such file or directory
Run actions/upload-artifact@v4
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
Uploaded bytes 14172
Finished uploading artifact content to blob storage!
SHA256 hash of uploaded artifact zip is d7eb3c26349191b249a257618b5500b3cba5b73da8b68678d310b2043e609835
Finalizing artifact upload
Artifact core-coverage-T2T1-core_device_test-1.zip successfully finalized. Artifact ID 2509055720
Artifact core-coverage-T2T1-core_device_test-1 has been successfully uploaded! Final size is 14172 bytes. Artifact ID is 2509055720
Artifact download URL: https://github.com/trezor/trezor-firmware/actions/runs/13047700328/artifacts/2509055720

Coverage report (T2T1)

Run nix-shell --run "poetry run make -C core coverage"
make: Entering directory '/home/runner/work/trezor-firmware/trezor-firmware/core'
./tools/coverage-report
COVERAGE_THRESHOLD set to 78%
.coverage.core_click_test0
.coverage.core_device_test1
.coverage.core_fido2_test0
.coverage.core_monero_test0
.coverage.core_persistence_test0
.coverage.core_u2f_test0
.coverage.gw0.core_device_test0
.coverage.gw1.core_device_test0
.coverage.gw2.core_device_test0
.coverage.gw3.core_device_test0
Creating .coverage.empty file
Combining .coverage.* files
SUCCESS: COVERAGE: 85%
See core/htmlcov/index.html for details.
make: Leaving directory '/home/runner/work/trezor-firmware/trezor-firmware/core'

@romanz
Copy link
Contributor Author

romanz commented Jan 30, 2025

Adding translations label to the PR, to test multiple Device tests jobs per model.

@romanz romanz added the translations Put this label on a PR to run tests in all languages label Jan 30, 2025
@romanz romanz force-pushed the romanz/coverage-tests branch 2 times, most recently from b2d3a0c to a14ead4 Compare January 30, 2025 09:08
@romanz romanz changed the base branch from main to romanz/update-fixtures January 30, 2025 09:10
@romanz
Copy link
Contributor Author

romanz commented Jan 30, 2025

Rebased over #4558 due to translated UI tests failure.

@romanz
Copy link
Contributor Author

romanz commented Jan 30, 2025

Coverage report seems to work with translations as well:

Coverage report (T2T1)

Run nix-shell --run "poetry run make -C core coverage"
make: Entering directory '/home/runner/work/trezor-firmware/trezor-firmware/core'
./tools/coverage-report
COVERAGE_THRESHOLD set to 78%
.coverage.core_click_test0
.coverage.core_click_test1
.coverage.core_click_test2
.coverage.core_click_test3
.coverage.core_click_test4
.coverage.core_click_test5
.coverage.core_device_test10
.coverage.core_device_test11
.coverage.core_device_test6
.coverage.core_device_test7
.coverage.core_device_test8
.coverage.core_device_test9
.coverage.core_fido2_test0
.coverage.core_monero_test0
.coverage.core_persistence_test0
.coverage.core_u2f_test0
.coverage.gw0.core_device_test0
.coverage.gw0.core_device_test1
.coverage.gw0.core_device_test2
.coverage.gw0.core_device_test3
.coverage.gw0.core_device_test4
.coverage.gw0.core_device_test5
.coverage.gw1.core_device_test0
.coverage.gw1.core_device_test1
.coverage.gw1.core_device_test2
.coverage.gw1.core_device_test3
.coverage.gw1.core_device_test4
.coverage.gw1.core_device_test5
.coverage.gw2.core_device_test0
.coverage.gw2.core_device_test1
.coverage.gw2.core_device_test2
.coverage.gw2.core_device_test3
.coverage.gw2.core_device_test4
.coverage.gw2.core_device_test5
.coverage.gw3.core_device_test0
.coverage.gw3.core_device_test1
.coverage.gw3.core_device_test2
.coverage.gw3.core_device_test3
.coverage.gw3.core_device_test4
.coverage.gw3.core_device_test5
Creating .coverage.empty file
Combining .coverage.* files
SUCCESS: COVERAGE: 85%
See core/htmlcov/index.html for details.
make: Leaving directory '/home/runner/work/trezor-firmware/trezor-firmware/core'

@romanz
Copy link
Contributor Author

romanz commented Jan 30, 2025

BTW, the issue reproduced here (before this fix is applied):
https://github.com/trezor/trezor-firmware/actions/runs/13049022289/job/36408662068?pr=4558
https://github.com/trezor/trezor-firmware/actions/runs/13049022289/job/36408662924?pr=4558

@romanz romanz requested a review from mmilata January 30, 2025 10:27
@romanz romanz mentioned this pull request Jan 30, 2025
Merged
@romanz romanz requested a review from matejcik January 30, 2025 10:30
@romanz romanz marked this pull request as ready for review January 30, 2025 10:31
mmilata
mmilata approved these changes Jan 30, 2025
View reviewed changes
Copy link
Member

@mmilata mmilata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this was bugging me for a long time but never realized it's because of a filename clash ... thanks!

Base automatically changed from romanz/update-fixtures to main January 30, 2025 12:26
@romanz
ci: fix JSONDecodeError when combining .coverage.gw* files
8083205 
For example: https://github.com/trezor/trezor-firmware/actions/runs/12976369614/job/36188913357

Single `.coverage` files were renamed, but `make test_emu_ui_multicore` generates
multiple `.coverage.gw*` files, which may overwrite each other in case there are
multiple similar CI jobs (e.g. when translation-related jobs are enabled).

[no changelog]
@romanz romanz force-pushed the romanz/coverage-tests branch from a14ead4 to 8083205 Compare January 30, 2025 12:45
@romanz
Copy link
Contributor Author

romanz commented Jan 30, 2025

Rebased:

$ git range-diff a14ead404^^..a14ead404 808320539^^..808320539
1:  e3b4775a6 = 1:  a7d7907ef chore: update translated UI fixtures
2:  a14ead404 = 2:  808320539 ci: fix JSONDecodeError when combining `.coverage.gw*` files

@romanz romanz merged commit a75a9f2 into main Jan 30, 2025
140 checks passed
@romanz romanz deleted the romanz/coverage-tests branch January 30, 2025 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mmilata mmilata mmilata approved these changes

@matejcik matejcik Awaiting requested review from matejcik

Assignees

@romanz romanz

Labels
translations Put this label on a PR to run tests in all languages
Projects
Firmware
Status: 🤝 Needs QA
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@romanz @mmilata