You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
A core problem of AI test generation is test quality itself. Is the test actually exercising the intended lines of code? Are the assertions correct (e.g., it's not just assert true and so on)? Many AI generated tests may be wrong, or just fail to pass. With AI Test Generation today, the only recourse the user has is to run the test generation process again and hope the result is better than the first run. However, this routinely results in poor outcomes since:
The context is the same as the first test generation run
The AI itself isn't informed of the failing test
However, given the data we collect on coverage reports and test results, Codecov is actually in a position to solve this.
Describe the solution you'd like
I'm proposing "Coverage informed Test generation". Consider the following process:
A user opens a PR.
The user asks Codecov AI to generate tests for the PR
The tests come back and they either pass or they fail.
If they fail, Codecov automatically sends the test failure reasons back to the AI, asking the AI to use the newly provided context on the previous tests failures and attempt to generate new tests. These new tests would be added as a new commit to the original PR.
If they pass, but Codecov's coverage check fails, the updated coverage information, tests, and diff will be sent to the AI to attempt to generate tests again to see if they pass the coverage check.
Steps 4 and/or 5 continue to run until test and coverage checks pass, or a sufficient number of retires (e.g., 3) has been performed.
This feature will help AI Test Generation generate more tests that pass and are actually useful for hitting coverage targets. This will allow even those teams who cannot dedicate meaningful resources to testing to begin the initial steps of building out a meaningful test suite.
Describe alternatives you've considered
The alternative here is really to do nothing and just hope AI Test Generation generates meaningful tests on its first attempt, or relying on the user to manually retry.
Additional context
If we build this we have to be mindful of CI costs incurred through reruns to generate passing tests, this is both in terms of money and time cost.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
A core problem of AI test generation is test quality itself. Is the test actually exercising the intended lines of code? Are the assertions correct (e.g., it's not just
assert true
and so on)? Many AI generated tests may be wrong, or just fail to pass. With AI Test Generation today, the only recourse the user has is to run the test generation process again and hope the result is better than the first run. However, this routinely results in poor outcomes since:However, given the data we collect on coverage reports and test results, Codecov is actually in a position to solve this.
Describe the solution you'd like
I'm proposing "Coverage informed Test generation". Consider the following process:
This feature will help AI Test Generation generate more tests that pass and are actually useful for hitting coverage targets. This will allow even those teams who cannot dedicate meaningful resources to testing to begin the initial steps of building out a meaningful test suite.
Describe alternatives you've considered
The alternative here is really to do nothing and just hope AI Test Generation generates meaningful tests on its first attempt, or relying on the user to manually retry.
Additional context
If we build this we have to be mindful of CI costs incurred through reruns to generate passing tests, this is both in terms of money and time cost.
The text was updated successfully, but these errors were encountered: