Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce regression tests with deterministic values #50

Closed
7 tasks done
CSchoel opened this issue Aug 11, 2024 · 1 comment · Fixed by #51 or #54
Closed
7 tasks done

Introduce regression tests with deterministic values #50

CSchoel opened this issue Aug 11, 2024 · 1 comment · Fixed by #51 or #54
Assignees

Comments

@CSchoel
Copy link
Owner

CSchoel commented Aug 11, 2024

Since we already know that the output of some metrics changes slightly when we switch to numpy 2.0 (see #42), we should protect against this early on. The best way I see to catch this is to have regression tests and check for exact results on some sample data.

  • Create a regression test branching out from main. Just use random data for each of the main algorithms.
  • (Optional) check if there are any major branches in any of the algorithms that only trigger under specific conditions that are not met with the random data. (We can use the debugger for that.)
  • (Optional) also test with algorithms that include randomness by keeping seed fixed.
  • Store the exact values produced by the algorithms on this data.
  • Document the tests.
  • Merge tests into dev branch.
  • Switch back to the poetry branch (using updated scikit-learn) and check if regression tests still run.
@CSchoel
Copy link
Owner Author

CSchoel commented Sep 30, 2024

I checked the regression tests with the following setups:

  • Python 3.12
    • scikit-learn==1.5.1
    • numpy==1.26.4
  • Python 3.8
    • scikit-learn==0.24.2
    • numpy==1.24.4
  • Python 3.7 (CI)
    • scikit_learn==1.0.2
    • numpy==1.21.6
  • Python 3.10 (CI)
    • scikit_learn==1.5.2
    • numpy==1.26.4

This should be good enough for now. The next step is to test what happens when we move to numpy 2.0.

@CSchoel CSchoel closed this as completed Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant