-
Notifications
You must be signed in to change notification settings - Fork 641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lucene.Net.TestFramework: Fixed random seed functionality so it is repeatable (Fixes #288) #547
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…st result on how to set a fixed random seed to reproduce the test result (pass or fail).
… and NUnitTestFixtureBuilder along with RandomSeedAttribute and LuceneRandomSeedInitializer to fix the broken randomized seed functionality so we can repeat test runs reliably.
… from a static field to a property and marked obsolete. Added a new property GoodFastHashSeed. Removed SystemProperties call to populate the value of the field, since NUnit only allows us to generate a seed per test, and we need a way to inject the seed value for repeatability.
…ded line to set StringHelper.goodFastHashSeed when the test framework is attached.
… test properties if we are in a suite context
…anged RandomSeedAttribute to be an assembly-level attribute
…and store the seed in a property of the test fixture (instance-based rather than static)
…usion for RandomSeed constant
…d system property to prefer "tests:culture" instead of "tests:locale" to set the culture, but left "tests:locale" in for backward/java compatibility.
… an NUnit test property for each test. This allows us to add additional strongly-typed properties to the test without wrapping the class. Changed to use J2N.Randomizer and changed all seeds to long rather than int. Converted the number for setting the random seed to hexadecimal and updated the message to add the setting for the current culture to the attributes/.runsettings file.
…lass or our default instance of LuceneTestFrameworkInitializer. Also added LuceneTestCase.SetUpFixture to control initialization of LuceneTestFrameworkInitializer so it is only called on setup and teardown for the assembly. Added Initialize() method to LuceneTestFrameworkInitializer that must be used to set the factories.
…iled random repeat message on failure or error
…tClassType() method and added TestType property
…ved LuceneTestCase parameter from Before() and After() methods.
…lizeTestFixture that accepts RandomizedContext so NUnitTestFixtureBuilder doesn't have to deal with setting NUnit properties. Also added RandomizedContext.CurrentTestAssembly property.
… a UseTempLineDocFileRule to scan for it and execute the file decompression and cleanup. Removed the unnecessary Startup.cs files that only served the purpose of getting NUnit to recognize LuceneTestFrameworkInitializer in the assembly scan.
…NUnit's [SetCulture] attribute to be used to set the culture
…ions to use NUnit's SetCulture attribute, since NUnit property attributes are not currently read by the test framework
…rom using System.Random to J2N.Randomizer
…ched from using System.Random to J2N.Randomizer. Also added feature to export a random SynonymMap as source code.
…ute and an issue URL.
…ance(): Ignore in net461, since it runs a bit slow in that environment.
…d [AwaitsFix] attribute, as this is sometimes failing (see apache#544)
… [AwaitsFix] attribute, since this test is failing on .NET Framework x86 in Release mode.
… APIs remain as System.Random, since J2N.Randomizer is a subclass.
…ance(): Adjusted limit from 2 min to 2.5 min for .NET Framework, since it runs a bit slow in CI.
NightOwl888
force-pushed
the
fix/GH-288
branch
from
November 22, 2021 22:28
34aedf0
to
08fbeec
Compare
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #288.
These changes are still experimental, have had minimal manual testing, and don't have any automated tests. But, the need to repeat a test failure outweighs the need to make these features stable before merging them, IMHO.
Features
OneTimeSetUp
andOneTimeTearDown
calls before the first test and after the last test in the fixture.LuceneTestCase.TestFixtureAttribute
- A copy of NUnit's attribute, modified with some extra hooks to setup our test framework.LuceneRandomSeedInitializer
- Determines whether to use a random seed or one from the current context to initialize the seeds for each test fixture/test. Also registers theRandomizedContext
for each test fixture/test.RandomizedContext
- A static class that is registered as an NUnit test property, which allows us to use strongly-typed properties forRandomSeed
(the initial seed),TestSeed
,CurrentTestAssembly
, and of course theRandomGenerator
for the context.LuceneTestCase.SetUpFixture
- An internal class that captures allOneTimeSetUp
andOneTimeTearDown
calls from our "wrapper" test fixture and uses reference counting to ensure we always callLuceneTestFrameworkInitializer
(or a subclass of it) exactly 1 time before the first test is run and after the last test is run.NUnitTestFixtureBuilder
- A copy of the class with the same name from NUnit, with the extra bits to inject the random seeds and take the filters out of the equation so the random generator gets the exact same number of calls regardless of which filters are applied. It also manages the assembly-levelRandomizedContext
instance, which is applied as a property to every "wrapper" test fixture.RandomSeedAttribute
- An assembly-level attribute that can be used to apply a random seed that was reported as a failure to repeat the results. It is also possible to set the random seed in a .runsettings file in the solution directory or alucene.testsettings.json
file in any directory in the path of the test assembly.UseTempLineDocsFileAttribute
- An assembly-level or class-level attribute that specifies to unzip theLineFileDocs
to a temp directory one time per assembly, so it doesn't have to be repeated in each test. This feature only works if the file has a.gz
extension. The path to the unzipped file is available from theLuceneTestCase.TestLineDocsFile
property. To specify an externalLineFileDocs
file, use thetests:linedocsfile
system property (which works from .runsettings or from lucene.testsettings.json).NUnit.Framework.SetCultureAttribute
- This attribute can be used to override the randomly generated culture. Or it can be set in .runsettings/lucene.testsettings.json using the keytests:culture
and the value of the culture (such asfr-CA
).Breaking Changes
Initialize()
virtual method was added toLuceneTestFrameworkInitializer
, which is where the propertiesCodecFactory
,DocValuesFormatFactory
,PostingsFormatFactory
, andConfigurationFactory
must be set. AnInvalidOperationException
will be thrown if they are set in the constructor,TestFrameworkSetUp()
orTestFrameworkTearDown()
.StringHelper.GOOD_FAST_HASH_SEED
has been turned into a static property and renamed toStringHelper.GoodFastHashSeed
(the seed is also a setting that is set to a fixed value during repeatable testing).LuceneTestCase.GetTestClass()
was deprecated and changed into a new propertyTestType
.AbstractBeforeAfterRule.Before()
andAbstractBeforeAfterRule.After()
methods have had theirLuceneTestCase
parameter removed.This PR also includes some plumbing to generate the source code to build a
SynonymMap
, so it can be quickly reconstituted for a non-random test (for internal debugging purposes).The
[AwaitsFix]
attribute has been applied to the remaining test failures, and theazure-pipelines.yml
is now set to fail the build if we get more than 0 test failures (previously this was set to 2).References