Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sanitycheck: skipped tests marked as failed due to the reason SKIPPED (SRAM overflow) #29412

Closed
maksimmasalski opened this issue Oct 21, 2020 · 0 comments · Fixed by #29717
Closed
Assignees
Labels
area: Sanitycheck Sanitycheck has been renamed to Twister bug The issue is a bug, or the PR is fixing a bug priority: medium Medium impact/importance bug

Comments

@maksimmasalski
Copy link
Collaborator

Describe the bug
On the Microchip board mec15xxevb_assy6853 I'm using Sanitycheck. For one test suite that should be skipped due to the SRAM overflow, sanitycheck marks it as failed.
When I run standalone sanitycheck for that test suite in the terminal I see Sanitycheck skipped it. When I go to check .xml file with the results, skipped tests marked as failed in the mec15xxevb_assy6853.xml file

To Reproduce
Steps to reproduce the behavior:

  1. sanitycheck --hardware-map ~/mec1.file --device-testing -T tests/portability/cmsis_rtos_v2/ -vv
    You will receive output, test skipped.
DEBUG   - Test skipped due to SRAM Overflow
INFO    - 1/1 mec15xxevb_assy6853       tests/portability/cmsis_rtos_v2/portability.cmsis_rtos_v2 SKIPPED (SRAM overflow)

DEBUG   - running footprint_reports
INFO    - 0 of 0 tests passed (0.00%), 0 failed, 1 skipped with 0 warnings in 6.92 seconds
INFO    - In total 0 test cases were executed on 1 out of total 297 platforms (0.34%)
INFO    - 0 tests executed on platforms, 0 tests were only built.
  1. Open sanity-out/sanitycheck.xml Result looks correct, skipped test suite marked as skipped
<testsuites><testsuite errors="0" failures="0" name="mec15xxevb_assy6853" skipped="1" tests="1" time="0.000000">
  1. Now open detailed description file sanity-out/mec15xxevb_assy6853.xml. You will see that you have failures! Tests were not skipped.
<testsuites><testsuite errors="0" failures="23" name="mec15xxevb_assy6853" skipped="0" tests="23" time="0.000000"><time="0.000000">

Expected behavior
Skipped tests must be skipped in all .xml files

Impact
Sanitycheck signals failure for the skipped tests. Not consistent QA results.

Environment (please complete the following information):
OS: Linux Ubuntu 18.04.4 LTS
Toolchain Zephyr SDK 0.11.4
Commit SHA or Version used 2a4d3bb

Additional context
I've tested with another test suite, that is going to be skipped, but due to another reason SKIPPED: Not runnable on device using same command:
sanitycheck --hardware-map ~/mec1.file --device-testing -T tests/drivers/spi/spi_loopback/ -vv

When I open sanity-out/sanitycheck.xml I can see correct behaviour, skipped tests detected:

<testsuites><testsuite errors="0" failures="0" name="mec15xxevb_assy6853" skipped="2" tests="2" time="0.000000">

When I open sanity-out/mec15xxevb_assy6853.xml I can see correct behaviour too, skipped tests detected:

<testsuites><testsuite errors="0" failures="0" name="mec15xxevb_assy6853" skipped="2" tests="2" time="0.000000">
@maksimmasalski maksimmasalski added the bug The issue is a bug, or the PR is fixing a bug label Oct 21, 2020
@maksimmasalski maksimmasalski added platform: Microchip MEC Microchip MEC Platform and removed platform: Microchip MEC Microchip MEC Platform labels Oct 21, 2020
@nashif nashif added priority: medium Medium impact/importance bug area: Sanitycheck Sanitycheck has been renamed to Twister labels Oct 21, 2020
nashif added a commit to nashif/zephyr that referenced this issue Nov 2, 2020
skips due to ram/rom overflow were captured as failures. Fix this and
count them correctly as skips.

Fixes zephyrproject-rtos#29412

Signed-off-by: Anas Nashif <[email protected]>
nashif added a commit that referenced this issue Nov 2, 2020
skips due to ram/rom overflow were captured as failures. Fix this and
count them correctly as skips.

Fixes #29412

Signed-off-by: Anas Nashif <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Sanitycheck Sanitycheck has been renamed to Twister bug The issue is a bug, or the PR is fixing a bug priority: medium Medium impact/importance bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants