Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to StableRNGs for tests #169

Merged
merged 5 commits into from
Jan 17, 2022
Merged

Conversation

kescobo
Copy link
Contributor

@kescobo kescobo commented Jan 14, 2022

See #168.

I just went ahead and did it, I hope that's OK 😳. I'm just really excited to see you working on this again, and want to help where I can. If you had other ideas, I won't be offended if you want to close this and do it elsewhere.

Aside from adding StableRNGs, also in this PR (easy to revert if you'd prefer) -

  • I removed REQUIRE - you're not testing on 0.7, and it seems like it would be hard to maintain that much backward compat.
  • Removed .travis.yml since you've got GHA going
  • I added a test/Project.toml so that tests can have their own environment, and you don't need to have StableRNGs.jl as a dependency in the main package itself.
  • I added 1.6 as an explicit test target. This is probably superfluous if you're testing on the current release and 1.1, but 🤷

To double check: There was one failing test in test/pca.jl that was due to the random numbers, but also one where I had to adjust the tolerance on isapprox() by a factor of 10 to make it pass (here). This doesn't seem like it should be related to the random number generation, but I don't know the method well enough to say.

One other thought: Given the way that the tests are currently set up, I wonder if you'd like to use ReTest.jl. I ask because, given the structure of the tests, I'm assuming that during development you're doing include(test/...) in the REPL. This can cause issues with global variables sticking around, and it means you have eg using Statistics over and over.

With ReTest.jl, you can explicitly filter which test sets run, only re-run tests that have changed etc. I recently switched my packages over and it's pretty easy. I'd be happy to do that here or in another PR if you're interested.

@wildart
Copy link
Collaborator

wildart commented Jan 15, 2022

Thanks for great work. There are several issues need to be addressed:

  1. You still need to add StableRNGs to extras & targets sections of Project.toml file, to avoid problems for older julia versions, see CI run.
[extras]
StableRNGs = "860ef19b-820b-49d6-a774-d7a799459cd3"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test", "StableRNGs"]
  1. isapprox calls should be changed to \approx binary operator.

This can cause issues with global variables sticking around, and it means you have eg using Statistics over and over.

Not really, tests for each algorithm are self-contained. Each method test is wrapped in testset which creates its own scope, preventing any variable leakage, and I do not mind of importing various packages all over again.

@kescobo
Copy link
Contributor Author

kescobo commented Jan 15, 2022

You still need to add StableRNGs to extras & targets sections of Project.toml file, to avoid problems for older julia versions, see CI run.

Oh, whoops - I thought 1.1 was when support for test/Project.toml came in (but it's actually from 1.2). If it's necessary to keep that stuff around anyway, perhaps I should ditch the test/Project.toml?

isapprox calls should be changed to \approx binary operator.

Not sure what you mean here - I don't think I introduced any new ones of these, and it can only change where atol is not needed, right? Or did you mean to change to eg ≈(x, y; atol=z)?

The only place I found where isappox() is used rather than in the absence of the atol kwarg is here. At least in the tests - there is also 1 place in src.

Not really, tests for each algorithm are self-contained. Each method test is wrapped in testset which creates its own scope, preventing any variable leakage, and I do not mind of importing various packages all over again.

Fair enough!

@wildart wildart merged commit 446239f into JuliaStats:master Jan 17, 2022
@wildart
Copy link
Collaborator

wildart commented Jan 17, 2022

Thanks. Great work.

@kescobo kescobo deleted the kescobo/tests branch January 18, 2022 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants