Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lucene.Net.TestFramework: Fixed random seed functionality so it is repeatable (Fixes #288) #547

Merged
merged 28 commits into from
Nov 22, 2021

Conversation

NightOwl888
Copy link
Contributor

@NightOwl888 NightOwl888 commented Nov 19, 2021

NOTE: This PR should be squashed.

Fixes #288.

These changes are still experimental, have had minimal manual testing, and don't have any automated tests. But, the need to repeat a test failure outweighs the need to make these features stable before merging them, IMHO.

Features

  • Random Seed Reporting - Upon a test failure, the random seed, random culture, and instructions on how to apply it are appended to the test message.
  • Test Fixture Wrapping - We add an extra outer nested test fixture to capture all of the OneTimeSetUp and OneTimeTearDown calls before the first test and after the last test in the fixture.
  • LuceneTestCase.TestFixtureAttribute - A copy of NUnit's attribute, modified with some extra hooks to setup our test framework.
  • LuceneRandomSeedInitializer - Determines whether to use a random seed or one from the current context to initialize the seeds for each test fixture/test. Also registers the RandomizedContext for each test fixture/test.
  • RandomizedContext - A static class that is registered as an NUnit test property, which allows us to use strongly-typed properties for RandomSeed (the initial seed), TestSeed, CurrentTestAssembly, and of course the RandomGenerator for the context.
  • LuceneTestCase.SetUpFixture - An internal class that captures all OneTimeSetUp and OneTimeTearDown calls from our "wrapper" test fixture and uses reference counting to ensure we always call LuceneTestFrameworkInitializer (or a subclass of it) exactly 1 time before the first test is run and after the last test is run.
  • NUnitTestFixtureBuilder - A copy of the class with the same name from NUnit, with the extra bits to inject the random seeds and take the filters out of the equation so the random generator gets the exact same number of calls regardless of which filters are applied. It also manages the assembly-level RandomizedContext instance, which is applied as a property to every "wrapper" test fixture.
  • RandomSeedAttribute - An assembly-level attribute that can be used to apply a random seed that was reported as a failure to repeat the results. It is also possible to set the random seed in a .runsettings file in the solution directory or a lucene.testsettings.json file in any directory in the path of the test assembly.
  • UseTempLineDocsFileAttribute - An assembly-level or class-level attribute that specifies to unzip the LineFileDocs to a temp directory one time per assembly, so it doesn't have to be repeated in each test. This feature only works if the file has a .gz extension. The path to the unzipped file is available from the LuceneTestCase.TestLineDocsFile property. To specify an external LineFileDocs file, use the tests:linedocsfile system property (which works from .runsettings or from lucene.testsettings.json).
  • NUnit.Framework.SetCultureAttribute - This attribute can be used to override the randomly generated culture. Or it can be set in .runsettings/lucene.testsettings.json using the key tests:culture and the value of the culture (such as fr-CA).

Breaking Changes

  • An Initialize() virtual method was added to LuceneTestFrameworkInitializer, which is where the properties CodecFactory, DocValuesFormatFactory, PostingsFormatFactory, and ConfigurationFactory must be set. An InvalidOperationException will be thrown if they are set in the constructor, TestFrameworkSetUp() or TestFrameworkTearDown().
  • StringHelper.GOOD_FAST_HASH_SEED has been turned into a static property and renamed to StringHelper.GoodFastHashSeed (the seed is also a setting that is set to a fixed value during repeatable testing).
  • LuceneTestCase.GetTestClass() was deprecated and changed into a new property TestType.
  • AbstractBeforeAfterRule.Before() and AbstractBeforeAfterRule.After() methods have had their LuceneTestCase parameter removed.

This PR also includes some plumbing to generate the source code to build a SynonymMap, so it can be quickly reconstituted for a non-random test (for internal debugging purposes).

The [AwaitsFix] attribute has been applied to the remaining test failures, and the azure-pipelines.yml is now set to fail the build if we get more than 0 test failures (previously this was set to 2).

References

…st result on how to set a fixed random seed to reproduce the test result (pass or fail).
… and NUnitTestFixtureBuilder along with RandomSeedAttribute and LuceneRandomSeedInitializer to fix the broken randomized seed functionality so we can repeat test runs reliably.
… from a static field to a property and marked obsolete. Added a new property GoodFastHashSeed. Removed SystemProperties call to populate the value of the field, since NUnit only allows us to generate a seed per test, and we need a way to inject the seed value for repeatability.
…ded line to set StringHelper.goodFastHashSeed when the test framework is attached.
… test properties if we are in a suite context
…anged RandomSeedAttribute to be an assembly-level attribute
…and store the seed in a property of the test fixture (instance-based rather than static)
…d system property to prefer "tests:culture" instead of "tests:locale" to set the culture, but left "tests:locale" in for backward/java compatibility.
… an NUnit test property for each test. This allows us to add additional strongly-typed properties to the test without wrapping the class. Changed to use J2N.Randomizer and changed all seeds to long rather than int. Converted the number for setting the random seed to hexadecimal and updated the message to add the setting for the current culture to the attributes/.runsettings file.
…lass or our default instance of LuceneTestFrameworkInitializer. Also added LuceneTestCase.SetUpFixture to control initialization of LuceneTestFrameworkInitializer so it is only called on setup and teardown for the assembly. Added Initialize() method to LuceneTestFrameworkInitializer that must be used to set the factories.
…iled random repeat message on failure or error
…tClassType() method and added TestType property
…ved LuceneTestCase parameter from Before() and After() methods.
…lizeTestFixture that accepts RandomizedContext so NUnitTestFixtureBuilder doesn't have to deal with setting NUnit properties. Also added RandomizedContext.CurrentTestAssembly property.
… a UseTempLineDocFileRule to scan for it and execute the file decompression and cleanup. Removed the unnecessary Startup.cs files that only served the purpose of getting NUnit to recognize LuceneTestFrameworkInitializer in the assembly scan.
…NUnit's [SetCulture] attribute to be used to set the culture
…ions to use NUnit's SetCulture attribute, since NUnit property attributes are not currently read by the test framework
…ched from using System.Random to J2N.Randomizer. Also added feature to export a random SynonymMap as source code.
…ance(): Ignore in net461, since it runs a bit slow in that environment.
…d [AwaitsFix] attribute, as this is sometimes failing (see apache#544)
… [AwaitsFix] attribute, since this test is failing on .NET Framework x86 in Release mode.
… APIs remain as System.Random, since J2N.Randomizer is a subclass.
…ance(): Adjusted limit from 2 min to 2.5 min for .NET Framework, since it runs a bit slow in CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix Random Seed Functionality in TestFramework
1 participant