Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spec11: harden testing #255

Merged
merged 4 commits into from
Dec 2, 2022
Merged

spec11: harden testing #255

merged 4 commits into from
Dec 2, 2022

Conversation

miri64
Copy link
Member

@miri64 miri64 commented Aug 2, 2022

This introduces several changes to the LoRaWAN tests, in the hopes to not make them fail that regular anymore:

  • Parametrize by DR (and the resulting tx_delay)
  • Add some grace time to the waiting between sending from the node and receiving at the TTN client
  • Mark task02 and task03 as flaky. Those regularly fail in the weekly builds. Marking them flaky will try to repeat them at least 3 in case of failure
  • Increase expect timeout on lorawan_tx for task03
  • Retry sending itself a few times for ABP (sometimes it just fails on the first time)
  • Remove ul_cnt setting. When set, TX for some reason never succeeds.

@miri64 miri64 requested a review from jia200x August 2, 2022 13:35
@miri64
Copy link
Member Author

miri64 commented Aug 2, 2022

There seems to be (appearently unrelated) issue where task 3 can't complete since the lorawan_tx fails. TTN does not show any packets for that, @jia200x is currently trying to figure out what went wrong.

@miri64 miri64 force-pushed the spec11/enh/parametrize-dr branch 8 times, most recently from bb3d89b to 96b0f19 Compare August 4, 2022 13:30
@miri64
Copy link
Member Author

miri64 commented Aug 4, 2022

It appears, the only reason https://github.com/RIOT-OS/RIOT/runs/7673103443?check_suite_focus=true fails now is because the experiment is suddenly stopped while the device tries to send. @aabadie @fjmolinas do you know if that is a known problem?

@miri64 miri64 mentioned this pull request Aug 4, 2022
88 tasks
@miri64 miri64 mentioned this pull request Aug 25, 2022
88 tasks
@miri64
Copy link
Member Author

miri64 commented Aug 26, 2022

From the discussion in #258 (comment), I was thinking: maybe it makes sense to additionally add a blacklist feature for nodes. I.e. if the test fails, add the capability to mark the failing node as defect, and rerun with another. I will investigate this.

@maribu maribu force-pushed the spec11/enh/parametrize-dr branch from 389f555 to 2c3c812 Compare December 2, 2022 09:55
@miri64 miri64 force-pushed the spec11/enh/parametrize-dr branch from 2c3c812 to ba6bd20 Compare December 2, 2022 09:59
@miri64 miri64 force-pushed the spec11/enh/parametrize-dr branch from ba6bd20 to c0774b6 Compare December 2, 2022 10:00
@miri64
Copy link
Member Author

miri64 commented Dec 2, 2022

Squashed.

@miri64 miri64 merged commit 16fd783 into master Dec 2, 2022
@MrKevinWeiss MrKevinWeiss deleted the spec11/enh/parametrize-dr branch November 6, 2023 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants