Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix code extractor to match elm-test-rs 2.x #38

Merged
merged 3 commits into from
Apr 12, 2022

Conversation

jiegillet
Copy link
Contributor

This solves #30 and #34.
Please check commits for details.

I didn't add tons of tests to test all of the branching, because I think the vast majority would be convoluted and unrealistic code.

This should be merged before #36, because it also introduces the new test name path "describe name 1 > describe name 2 > ... > test name".

Solves exercism#30.
The code extractor can now:
- Find test/describe inside of a let declaration
- Find test/describe pretty much anywhere (if, record, tuple...)
- Handle more than one top-level `Test`
- Find tests in arbitrarily nested describe
- Write test name as "describe name 1 > describe name 2 > ... > test name"
@jiegillet jiegillet added x:module/test-runner Work on Test Runners x:type/coding Write code that is not student-facing content (e.g. test-runners, generators, but not exercises) x:size/medium Medium amount of work labels Apr 7, 2022
@jiegillet jiegillet requested a review from a team as a code owner April 7, 2022 12:47
@jiegillet jiegillet changed the title Fix code extractor to match elm-test-rs 2.0.1 Fix code extractor to match elm-test-rs 2.x Apr 7, 2022
@ErikSchierboom
Copy link
Member

Are there tests to verify that this doesn't break anything?

@jiegillet
Copy link
Contributor Author

Heh, all the tests are in the other PR :p
It shouldn't change anything for the current setting, but I can run a few sanity checks tomorrow.

@ErikSchierboom
Copy link
Member

👍

@mpizenberg
Copy link
Member

mpizenberg commented Apr 7, 2022

Thanks @jiegillet could you explain the main changes introduced in the extracting code to handle these situations?

Also, regarding the current limitations, it is nice to better handle test expressions that can start with other things than test "name" <| but it is still valuable to forbid those if possible when creating test files.

For example, in case of usage of let value = ... in (test "name " <| \_ -> -- using value for expectation) if we only extract the part with the expectation from the test, the student will not know what value is equal to when the test is run. This is especially important for concept exercises where students do not have access to the test file and rely on what is displayed on the website interface.

Similarly, any kind of expression that results in unknowns when considering only the part of the code with the expectations (the snippet) can cause some confusion for students if they don't have the whole context in the test snippet.

So I'm wondering if instead of trying to handle those situations, we might want instead to output some keywords that our CI can spot and report to fail before merging such tests with ambiguous snippets extractions.

@mpizenberg
Copy link
Member

Considering the comment above, it might be worth splitting this PR in two. (1) the part that updates the test name to be unique by taking into account the description hierarchy. And (2) that improves the snippet extractor capabilities.

(1) is unambiguously good and needed before merging the 2.0.1 elm-test-rs update.

(2) might need some more discussions

@jiegillet
Copy link
Contributor Author

Considering the comment above, it might be worth splitting this PR in two. (1) the part that updates the test name to be unique by taking into account the description hierarchy. And (2) that improves the snippet extractor capabilities.

Alright. For this PR, (1) and (2) are deeply intertwined, so I reverted to the original version and did the minimal changes to achieve (1).
@ErikSchierboom this change will not make any difference to the way tests are currently running, it's only updating the name of the test code snippets, and that doesn't influence the final result at all because the field is actually ignored (it shouldn't be, but that is fixed in the other PR).

For the rest, I do agree that we should have tests that capture as much information as possible in the body of the test, especially for concept exercises. We could have CI checks, but ultimately it seems to me that code review is the most important tool we have. However, as we repeatedly saw with robot-simulator, when review fails (or for old exercises), it is important to have code that is flexible enough to handle imperfect tests instead of crashing.

So in my opinion (in the future), we should

  1. Update the elm documentation to be a lot more clear about test expectations/limitations
  2. Have code extracting code that goes beyond the expectations anyway
  3. Add CI checks (we can run elm-test-runner in the elm CI, maybe with specific checks)

Copy link
Member

@mpizenberg mpizenberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@jiegillet
Copy link
Contributor Author

@ErikSchierboom could you merge this?

@ErikSchierboom ErikSchierboom merged commit 0528d8e into exercism:main Apr 12, 2022
@ErikSchierboom
Copy link
Member

Sure!

@jiegillet jiegillet deleted the jie-test-names branch April 13, 2022 01:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
x:module/test-runner Work on Test Runners x:size/medium Medium amount of work x:type/coding Write code that is not student-facing content (e.g. test-runners, generators, but not exercises)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants