Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage informed Test Generation. #3220

Open
eliatcodecov opened this issue Jan 14, 2025 · 0 comments
Open

Coverage informed Test Generation. #3220

eliatcodecov opened this issue Jan 14, 2025 · 0 comments

Comments

@eliatcodecov
Copy link

Is your feature request related to a problem? Please describe.
A core problem of AI test generation is test quality itself. Is the test actually exercising the intended lines of code? Are the assertions correct (e.g., it's not just assert true and so on)? Many AI generated tests may be wrong, or just fail to pass. With AI Test Generation today, the only recourse the user has is to run the test generation process again and hope the result is better than the first run. However, this routinely results in poor outcomes since:

  1. The context is the same as the first test generation run
  2. The AI itself isn't informed of the failing test

However, given the data we collect on coverage reports and test results, Codecov is actually in a position to solve this.

Describe the solution you'd like

I'm proposing "Coverage informed Test generation". Consider the following process:

  1. A user opens a PR.
  2. The user asks Codecov AI to generate tests for the PR
  3. The tests come back and they either pass or they fail.
  4. If they fail, Codecov automatically sends the test failure reasons back to the AI, asking the AI to use the newly provided context on the previous tests failures and attempt to generate new tests. These new tests would be added as a new commit to the original PR.
  5. If they pass, but Codecov's coverage check fails, the updated coverage information, tests, and diff will be sent to the AI to attempt to generate tests again to see if they pass the coverage check.
  6. Steps 4 and/or 5 continue to run until test and coverage checks pass, or a sufficient number of retires (e.g., 3) has been performed.

This feature will help AI Test Generation generate more tests that pass and are actually useful for hitting coverage targets. This will allow even those teams who cannot dedicate meaningful resources to testing to begin the initial steps of building out a meaningful test suite.

Describe alternatives you've considered
The alternative here is really to do nothing and just hope AI Test Generation generates meaningful tests on its first attempt, or relying on the user to manually retry.

Additional context
If we build this we have to be mindful of CI costs incurred through reruns to generate passing tests, this is both in terms of money and time cost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant