Backport 3.6: Test cases not executed: switch to enforcement mode #9680
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This clears more than half of the test cases that are not executed. This also captures a few negative test cases that are executed. Subsequent commits will refine the filtering. Signed-off-by: Gilles Peskine <[email protected]>
… tests Some negative tests involving unsupported mechanisms are executed, because they're testing what happens if the mechanism is unsupported. Refine the ignore list for `test_suite_psa_crypto_generate_key.generated` and `test_suite_psa_crypto_op_fail.generated` accordingly. Signed-off-by: Gilles Peskine <[email protected]>
…aised Ignore certain test cases which either should not be generated or should be executed. For each ignore list entry, link to a GitHub issue whose definition of done includes removing the entry. Signed-off-by: Gilles Peskine <[email protected]>
Our PSA crypto implementation does not and will not support SECP224K1. Signed-off-by: Gilles Peskine <[email protected]>
For each ignore list entry, link to a GitHub issue for its resolution, except for ssl-opt Valgrind tests which we never intend to run on the CI. Signed-off-by: Gilles Peskine <[email protected]>
For each ignore list entry, link to a GitHub issue for its resolution. Signed-off-by: Gilles Peskine <[email protected]>
For each ignore list entry, link to a GitHub issue for its resolution, except for a few configurations which there is a good reason to leave uncovered. Signed-off-by: Gilles Peskine <[email protected]>
Signed-off-by: Gilles Peskine <[email protected]>
MBEDTLS_ECP_NO_FALLBACK is only relevant when an ALT implementation of ECP is used. We don't test ALT implementations so we don't test MBEDTLS_ECP_NO_FALLBACK either. Signed-off-by: Gilles Peskine <[email protected]>
Signed-off-by: Gilles Peskine <[email protected]>
gilles-peskine-arm
added
needs-ci
This was referenced Oct 10, 2024
Signed-off-by: Gilles Peskine <[email protected]>
gilles-peskine-arm
force-pushed
the
test-cases-not-executed-enforce-3.6
branch
from
71341c6 to
8f88645
Compare
In 3.6, compared with the state of development that this work was based on, we're doing more testing with TLS12_PRF and TLS12_PSK_TO_MS disabled, thanks to some configurations with MBEDTLS_PSA_CRYPTO_CONFIG disabled. Signed-off-by: Gilles Peskine <[email protected]>
gilles-peskine-arm
added
needs-review
Signed-off-by: Gilles Peskine <[email protected]>
yanesca
previously approved these changes
Oct 11, 2024
gilles-peskine-arm
added
approved
Signed-off-by: Gilles Peskine <[email protected]>
gilles-peskine-arm
dismissed stale reviews from paul-elliott-arm and yanesca
via
6d268e3
gilles-peskine-arm
added
needs-review
…ed-enforce-3.6 Conflicts: * framework: update to the head of 'main'.
gilles-peskine-arm
dismissed stale reviews from paul-elliott-arm and yanesca
via
2d94866
yanesca
approved these changes
Oct 18, 2024
paul-elliott-arm
added
approved
Merged
via the queue into
Mbed-TLS:mbedtls-3.6
with commit Oct 18, 2024
043f6dc
4 of 5 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Switch the test coverage analysis in
analyze_outcomes.pyto enforcement mode. In other words, the CI will now complain if there is a test case that is not executed in any configuration. Legitimate exceptions can be recorded in the script.This is a backport of #9593. Most commits are cherry-picked, but there are a few differences due to how development has already diverged from 3.6.
PR checklist