Add golden test machinery #311
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR adds a new GoldenTestSpec, which exposes a `golden` combinator for building golden tests. These tests take a SymbolicObj3, and will raytrace it, checking the resulting STL against a known-good version. The result is a robust system for ensuring mesh generation doesn't subtly change when working on the underlying infrastructure.
julialongtin
approved these changes
Nov 24, 2020
|
Excellent! In that case I'll do some more work on it tomorrow to get everything nicely polished here. |
|
@julialongtin PTAL |
|
I plan to follow up this PR with another that copies the haskell examples in as golden tests. |
julialongtin
approved these changes
Nov 26, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new GoldenTestSpec, which exposes a
goldencombinator for building golden tests. These tests take a SymbolicObj3, and will raytrace it, checking the resulting STL against a known-good version. The result is a robust system for ensuring mesh generation doesn't subtly change when working on the underlying infrastructure.@julialongtin, you seemed somewhat against golden tests in our brief discussion on IRC. I'd urge you to reconsider; checking in the test result artifacts has the benefit that changes which (intentionally or otherwise) modify the mesh generation must come with a corresponding diff to the tests. This gives at-a-glance review-time reassurance that no bugs have been smuggled in.
Furthermore, it reifies your testing strategy of "check out the old branch, build some STLs, hash them, check out the new branch, build the same STLs, and compare the hashes" --- allowing other contributors to share that testing burden with you (albeit in an automated way.) The current makefile-based solution doesn't work at all for stack-managed GHC installs, and as we've seen on the GHC codebase, this is a surprisingly difficult problem to get right. Simpler for everyone to just put it into Haskell.