Test mechanism to automatically rewrite baselines in source code #26723
Conversation
|
The only drawback is (and was in past which caused to avoid doing this on the fly), is this works fine for handful of test. For large number of tests, you would be rewriting whole source code with one updated AssertSql - number of test failed in that collection. Pus code also uses lock so it makes it sequential across all the test failures. I don't have anything against putting this code in as it is test code only. |
Well, in the normal case, xunit doesn't parallelize tests in the same class (roughly file), right? Are you saying you think this approach has a perf problem because it parses and rewrites the same large source file? Intuitively this seems like it should be negligible to me, unless we're dealing with really huge files (we could run a quick extreme test by making all SQL assertions fail in some way). If it really does turn out to be a problem, there may also be a way to cache stuff in memory. |
|
Ah, what we could do is lock per file instead of globally - this makes sense. This way parallel tests running in different files can update concurrently. |
|
Another idea is to cache the Roslyn syntax tree for each file, and replace the old method invocation with the new. This would at least obviate having to reparse the file each time with Roslyn. We could similarly also cache the file as a string to avoid having to read it each time. I'm still not sure this is worth the trouble - it seems acceptable to wait a bit when running the test suite with lots of assertion failures. But if you think it's important, I can make those changes. |
|
Also update CONTRIBUTING.md |
roji
force-pushed
the
Superb
branch
2 times, most recently
from
64e055d to
7a66780
Compare
If the environment variable EF_TEST_REWRITE_BASELINES is set to 1, any SQL assertion failure will cause the baseline to be updated to the new baseline in the original source file.
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…net#26723) If the environment variable EF_TEST_REWRITE_BASELINES is set to 1, any SQL assertion failure will cause the baseline to be updated to the new baseline in the original source file.
Team, I give you SUPERB - the Supreme Ultimate Programmatic Enhancement for Rewriting Baselines.
If the environment variable EF_TEST_REWRITE_BASELINES is set to 1, any SQL assertion failure will cause the baseline to be updated to the new baseline in the original source file. This is implemented by using Roslyn to parse the source and find the AssertSql invocation expression, which is then replaced. I have nothing against reptile-based methods, but this seems more reliable and doesn't require any extra steps.
Just set the env var, make some huge change and see how it ripples by looking at the git diff!