Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional mutant statuses (TIMED_OUT, BUILD_FAILED) #40

Closed
k3rn31 opened this issue Jul 22, 2022 · 1 comment · Fixed by #49
Closed

Additional mutant statuses (TIMED_OUT, BUILD_FAILED) #40

k3rn31 opened this issue Jul 22, 2022 · 1 comment · Fixed by #49
Assignees
Labels
c/feature Category: An issue or PR related to a new feature
Milestone

Comments

@k3rn31
Copy link
Member

k3rn31 commented Jul 22, 2022

Currently, Gremlins supports KILLED, LIVED, NOT_COVERED and RUNNABLE status.
It should also report if the test fails for a timeout or for a build error, so two new statuses should be introduced:

  • TIMED_OUT
  • BUILD_FAILED

BUILD_FAILED

Should be easy enough to implement: tests failed for a build error have exit code 2 (while test failures have 1). We can simply set the mutation status based on the exit code of the tests.

TIMED_OUT

This is trickier because timed out tests have exit code 1 (same as normal test failures). How can detect if the test has timed out?
I can think of two options:

  1. Search the output of the test looking for "panic: test timed out after XXs" string (ugly solution)
  2. Do not use the -timeout XXs flag in the test and manage test timeouts outside of the command, using a context with timeout and exec.CommandContext,

Ideas are welcome.

@k3rn31 k3rn31 added the c/feature Category: An issue or PR related to a new feature label Jul 22, 2022
@k3rn31 k3rn31 added this to the v0.2.0 milestone Jul 22, 2022
@k3rn31 k3rn31 changed the title Additional mutant statuses Additional mutant statuses (TIMED_OUT, BUILD_FAILED) Jul 22, 2022
@k3rn31 k3rn31 modified the milestones: v0.2.0, v0.1.0 Jul 22, 2022
@k3rn31 k3rn31 self-assigned this Jul 22, 2022
k3rn31 added a commit that referenced this issue Jul 23, 2022
@k3rn31 k3rn31 linked a pull request Jul 23, 2022 that will close this issue
@k3rn31
Copy link
Member Author

k3rn31 commented Jul 23, 2022

NOT VIABLE is a better name than BUILD FAILED, since it remains more generic and can be used for other use cases in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/feature Category: An issue or PR related to a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant