sanitycheck: skipped tests marked as failed due to the reason SKIPPED (SRAM overflow) #29412
Labels
area: Sanitycheck
Sanitycheck has been renamed to Twister
bug
The issue is a bug, or the PR is fixing a bug
priority: medium
Medium impact/importance bug
Describe the bug
On the Microchip board mec15xxevb_assy6853 I'm using Sanitycheck. For one test suite that should be skipped due to the SRAM overflow, sanitycheck marks it as failed.
When I run standalone sanitycheck for that test suite in the terminal I see Sanitycheck skipped it. When I go to check .xml file with the results, skipped tests marked as failed in the
mec15xxevb_assy6853.xml
fileTo Reproduce
Steps to reproduce the behavior:
You will receive output, test skipped.
sanity-out/sanitycheck.xml
Result looks correct, skipped test suite marked as skippedsanity-out/mec15xxevb_assy6853.xml
. You will see that you have failures! Tests were not skipped.Expected behavior
Skipped tests must be skipped in all .xml files
Impact
Sanitycheck signals failure for the skipped tests. Not consistent QA results.
Environment (please complete the following information):
OS: Linux Ubuntu 18.04.4 LTS
Toolchain Zephyr SDK 0.11.4
Commit SHA or Version used 2a4d3bb
Additional context
I've tested with another test suite, that is going to be skipped, but due to another reason
SKIPPED: Not runnable on device
using same command:sanitycheck --hardware-map ~/mec1.file --device-testing -T tests/drivers/spi/spi_loopback/ -vv
When I open
sanity-out/sanitycheck.xml
I can see correct behaviour, skipped tests detected:When I open
sanity-out/mec15xxevb_assy6853.xml
I can see correct behaviour too, skipped tests detected:The text was updated successfully, but these errors were encountered: