Lucene.Net.TestFramework: Fixed random seed functionality so it is re…

…peatable (Fixes #288) (#547)

* Lucene.Net.TestFramework.Util.LuceneTestCase: Added message to the test result on how to set a fixed random seed to reproduce the test result (pass or fail).

* Lucene.Net.TestFramework.Util: Added customized TestFixtureAttribute, and NUnitTestFixtureBuilder along with RandomSeedAttribute and LuceneRandomSeedInitializer to fix the broken randomized seed functionality so we can repeat test runs reliably.

* BREAKING: Lucene.Net.Util.StringHelper.GOOD_FAST_HASH_SEED: converted from a static field to a property and marked obsolete. Added a new property GoodFastHashSeed. Removed SystemProperties call to populate the value of the field, since NUnit only allows us to generate a seed per test, and we need a way to inject the seed value for repeatability.

* Lucene.Net.TestFramework.Support.Util.LuceneRandomSeedInitializer: Added line to set StringHelper.goodFastHashSeed when the test framework is attached.

* Lucene.Net.TestFramework.Util.LuceneTestCase: Stow Random instance in test properties if we are in a suite context

* Lucene.Net.TestFramework: Reworked LuceneRandomSeedIntiailizer and changed RandomSeedAttribute to be an assembly-level attribute

* Lucene.Net.TestFramework: Removed support for Randomizer.InitialSeed and store the seed in a property of the test fixture (instance-based rather than static)

* Lucene.Net.Tests.TestFramework.Support.TestApiConsistency: Added exclusion for RandomSeed constant

* Lucene.Net.TestFramework.Util.TestRuleSetupAndRestoreClassEnv: Changed system property to prefer "tests:culture" instead of "tests:locale" to set the culture, but left "tests:locale" in for backward/java compatibility.

* Lucene.Net.TestFramework: Created RandomizedContext class to store as an NUnit test property for each test. This allows us to add additional strongly-typed properties to the test without wrapping the class. Changed to use J2N.Randomizer and changed all seeds to long rather than int. Converted the number for setting the random seed to hexadecimal and updated the message to add the setting for the current culture to the attributes/.runsettings file.

* BREAKING: Added LuceneSetUpFixtureBuilder class to load either a subclass or our default instance of LuceneTestFrameworkInitializer. Also added LuceneTestCase.SetUpFixture to control initialization of LuceneTestFrameworkInitializer so it is only called on setup and teardown for the assembly. Added Initialize() method to LuceneTestFrameworkInitializer that must be used to set the factories.

* Lucene.Net.TestFramework.Util.LuceneTestCase: Only print out the detailed random repeat message on failure or error

* BREAKING: Lucene.Net.TestFramework.Util.LuceneTestCase: Deprecated GetClassType() method and added TestType property

* BREAKING: Lucene.Net.TestFramework.Util.AbstractBeforeAfterRule: Removed LuceneTestCase parameter from Before() and After() methods.

* Lucene.Net.Util.LuceneRandomSeedInitializer: Added overload of InitializeTestFixture that accepts RandomizedContext so NUnitTestFixtureBuilder doesn't have to deal with setting NUnit properties. Also added RandomizedContext.CurrentTestAssembly property.

* Lucene.Net.TestFramework.Util: Added UseTempLineDocsFileAttribute and a UseTempLineDocFileRule to scan for it and execute the file decompression and cleanup. Removed the unnecessary Startup.cs files that only served the purpose of getting NUnit to recognize LuceneTestFrameworkInitializer in the assembly scan.

* Lucene.Net.TestFramework.Util.TestRuleSetupAndRestoreClassEnv: Allow NUnit's [SetCulture] attribute to be used to set the culture

* Lucene.Net.TestFramework.Util.LuceneTestCase: Updated repeat instructions to use NUnit's SetCulture attribute, since NUnit property attributes are not currently read by the test framework

* Lucene.Net.TestFramework.Analysis.BaseTokenStreamTestCase: Switched from using System.Random to J2N.Randomizer

* Lucene.Net.Tests.Analysis.Common.Analysis.Core.TestRandomChains: Switched from using System.Random to J2N.Randomizer. Also added feature to export a random SynonymMap as source code.

* azure-pipelines.yml: set maximumAllowedFailures to 0

* SWEEP: Marked all of the latest test failures with [AwaitsFix] attribute and an issue URL.

* Lucene.Net.Classification.KNearestNeighborClassifierTest::TestPerformance(): Ignore in net461, since it runs a bit slow in that environment.

* Lucene.Net.Index.TestIndexWriter::TestThreadInterruptDeadlock(): Added [AwaitsFix] attribute, as this is sometimes failing (see #544)

* Lucene.Net.Tests.Join.TestJoinUtil::TestMultiValueRandomJoin(): Added [AwaitsFix] attribute, since this test is failing on .NET Framework x86 in Release mode.

* SWEEP: Changed all test instances of System.Random to J2N.Randomizer. APIs remain as System.Random, since J2N.Randomizer is a subclass.

* Lucene.Net.TestFramework.Util.MethodInfoComparer: Made into singleton

* Lucene.Net.Classification.KNearestNeighborClassifierTest::TestPerformance(): Adjusted limit from 2 min to 2.5 min for .NET Framework, since it runs a bit slow in CI.

azure-pipelines.yml

@@ -336,17 +336,17 @@ stages:
           osName: 'Windows'
           imageName: 'windows-2019'
           maximumParallelJobs: 8
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         Linux:
           osName: 'Linux'
           imageName: 'ubuntu-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         macOS:
           osName: 'macOS'
           imageName: 'macOS-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
     displayName: 'Test net6.0,x64 on'
     pool:
       vmImage: $(imageName)
@@ -370,17 +370,17 @@ stages:
           osName: 'Windows'
           imageName: 'windows-2019'
           maximumParallelJobs: 8
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         Linux:
           osName: 'Linux'
           imageName: 'ubuntu-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         macOS:
           osName: 'macOS'
           imageName: 'macOS-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
     displayName: 'Test net6.0,x86 on'
     pool:
       vmImage: $(imageName)
@@ -404,17 +404,17 @@ stages:
           osName: 'Windows'
           imageName: 'windows-2019'
           maximumParallelJobs: 8
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         Linux:
           osName: 'Linux'
           imageName: 'ubuntu-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         macOS:
           osName: 'macOS'
           imageName: 'macOS-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
     displayName: 'Test net5.0,x64 on'
     pool:
       vmImage: $(imageName)
@@ -438,17 +438,17 @@ stages:
           osName: 'Windows'
           imageName: 'windows-2019'
           maximumParallelJobs: 8
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         Linux:
           osName: 'Linux'
           imageName: 'ubuntu-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         macOS:
           osName: 'macOS'
           imageName: 'macOS-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
     displayName: 'Test net5.0,x86 on'
     pool:
       vmImage: $(imageName)
@@ -472,17 +472,17 @@ stages:
           osName: 'Windows'
           imageName: 'windows-2019'
           maximumParallelJobs: 8
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         Linux:
           osName: 'Linux'
           imageName: 'ubuntu-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         macOS:
           osName: 'macOS'
           imageName: 'macOS-10.15' # macOS-latest should not be used here because we may get OS Darwin 19.6.0, which isn't supported fully (we get PlatformNotSupportedException).
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
     displayName: 'Test netcoreapp3.1,x64 on'
     pool:
       vmImage: $(imageName)
@@ -506,17 +506,17 @@ stages:
           osName: 'Windows'
           imageName: 'windows-2019'
           maximumParallelJobs: 8
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         Linux:
           osName: 'Linux'
           imageName: 'ubuntu-latest'
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
         macOS:
           osName: 'macOS'
           imageName: 'macOS-10.15' # macOS-latest should not be used here because we may get OS Darwin 19.6.0, which isn't supported fully (we get PlatformNotSupportedException).
           maximumParallelJobs: 7
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
     displayName: 'Test netcoreapp3.1,x86 on'
     pool:
       vmImage: $(imageName)
@@ -540,7 +540,7 @@ stages:
           osName: 'Windows'
           imageName: 'windows-2019'
           maximumParallelJobs: 8
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
     displayName: 'Test net461,x64 on'
     pool:
       vmImage: $(imageName)
@@ -564,7 +564,7 @@ stages:
           osName: 'Windows'
           imageName: 'windows-2019'
           maximumParallelJobs: 8
-          maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+          maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
     displayName: 'Test net461,x86 on'
     pool:
       vmImage: $(imageName)
@@ -595,7 +595,7 @@ stages:
         nugetArtifactName: '$(NuGetArtifactName)'
         testResultsArtifactName: '$(TestResultsArtifactName)'
         maximumParallelJobs: 8
-        maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+        maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
 
   - job: Test_net48_Windows_x86 # Only run Nightly or if explicitly enabled with RunX86Tests
     condition: and(succeeded(), ne(variables['RunTests'], 'false'), or(eq(variables['IsNightly'], 'true'), eq(variables['RunX86Tests'], 'true')))
@@ -612,7 +612,7 @@ stages:
         nugetArtifactName: '$(NuGetArtifactName)'
         testResultsArtifactName: '$(TestResultsArtifactName)'
         maximumParallelJobs: 8
-        maximumAllowedFailures: 2 # Maximum allowed failures for a successful build
+        maximumAllowedFailures: 0 # Maximum allowed failures for a successful build
 
 
 - stage: Publish_Stage

src/Lucene.Net.Benchmark/ByTask/Feeds/DocMaker.cs

@@ -505,7 +505,7 @@ public virtual void SetConfig(Config config, ContentSource source)
             updateDocIDLimit = config.Get("doc.random.id.limit", -1);
             if (updateDocIDLimit != -1)
             {
-                r = new Random(179);
+                r = new J2N.Randomizer(179);
             }
         }
     }

src/Lucene.Net.Benchmark/ByTask/Feeds/RandomFacetSource.cs

@@ -95,7 +95,7 @@ protected override void Dispose(bool disposing)
         public override void SetConfig(Config config)
         {
             base.SetConfig(config);
-            random = new Random(config.Get("rand.seed", 13));
+            random = new J2N.Randomizer(config.Get("rand.seed", 13));
             maxDocFacets = config.Get("max.doc.facets", 10);
             maxDims = config.Get("max.doc.facets.dims", 5);
             maxFacetDepth = config.Get("max.facet.depth", 3);

src/Lucene.Net.Benchmark/ByTask/Feeds/SortableSingleDocSource.cs

@@ -109,7 +109,7 @@ public override void SetConfig(Config config)
         {
             base.SetConfig(config);
             sortRange = config.Get("sort.rng", 20000);
-            r = new Random(config.Get("rand.seed", 13));
+            r = new J2N.Randomizer(config.Get("rand.seed", 13));
         }
     }
 }

src/Lucene.Net.Benchmark/ByTask/Feeds/SpatialDocMaker.cs

@@ -219,7 +219,7 @@ public IShape Convert(IShape shape)
                     double radius = radiusDegrees;
                     if (plusMinus > 0.0)
                     {
-                        Random random = new Random(point.GetHashCode());//use hashCode so it's reproducibly random
+                        Random random = new J2N.Randomizer(point.GetHashCode());//use hashCode so it's reproducibly random
                         radius += random.NextDouble() * 2 * plusMinus - plusMinus;
                         radius = Math.Abs(radius);//can happen if configured plusMinus > radiusDegrees
                     }

src/Lucene.Net.Expressions/JS/JavascriptCompiler.cs

@@ -194,7 +194,7 @@ private Expression CompileExpression()
 
         private void BeginCompile()
         {
-            var assemblyName = new AssemblyName("Lucene.Net.Expressions.Dynamic" + new Random().Next());
+            var assemblyName = new AssemblyName("Lucene.Net.Expressions.Dynamic" + Math.Abs(new J2N.Randomizer().Next()));
             asmBuilder = AssemblyBuilder.DefineDynamicAssembly(assemblyName, AssemblyBuilderAccess.RunAndCollect);
 
             modBuilder = asmBuilder.DefineDynamicModule(assemblyName.Name + ".dll");

src/Lucene.Net.Suggest/Suggest/UnsortedInputIterator.cs

@@ -43,7 +43,7 @@ public UnsortedInputEnumerator(IInputEnumerator source)
             : base(source)
         {
             ords = new int[m_entries.Length];
-            Random random = new Random();
+            Random random = new J2N.Randomizer();
             for (int i = 0; i < ords.Length; i++)
             {
                 ords[i] = i;

src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs

@@ -95,6 +95,8 @@ public override void CopyTo(IAttribute target)
     /// </summary>
     /// <seealso cref="MockAnalyzer"/>
     /// <seealso cref="MockTokenizer"/>
+    // LUCENENET specific - Specify to unzip the line file docs
+    [UseTempLineDocsFile]
     public abstract class BaseTokenStreamTestCase : LuceneTestCase
 #if TESTFRAMEWORK_XUNIT
         , Xunit.IClassFixture<BeforeAfterClass>
@@ -126,13 +128,6 @@ public BaseTokenStreamTestCase(BeforeAfterClass beforeAfter)
 
         // LUCENENET specific - de-nested CheckClearAttributesAttribute
 
-        // LUCENENET specific - Specify to unzip the line file docs
-        public override void BeforeClass()
-        {
-            UseTempLineDocsFile = true;
-            base.BeforeClass();
-        }
-
         // offsetsAreCorrect also validates:
         //   - graph offsets are correct (all tokens leaving from
         //     pos X have the same startOffset; all tokens
@@ -715,7 +710,7 @@ public override void Run()
                     if (latch != null) latch.Wait();
                     // see the part in checkRandomData where it replays the same text again
                     // to verify reproducability/reuse: hopefully this would catch thread hazards.
-                    CheckRandomData(new Random((int)seed), a, iterations, maxWordLength, useCharFilter, simple, offsetsAreCorrect, iw);
+                    CheckRandomData(new J2N.Randomizer(seed), a, iterations, maxWordLength, useCharFilter, simple, offsetsAreCorrect, iw);
                     success = true;
                 }
                 catch (Exception e) when (e.IsException()) // LUCENENET TODO: This catch block can be removed because Rethrow.rethrow() simply does what its name says, but need to get rid of the FirstException functionality and fix ThreadJob so it re-throws ThreadInterruptedExcepetion first.
@@ -754,7 +749,7 @@ public void CheckRandomData(Random random, Analyzer a, int iterations, int maxWo
 #endif
         {
             CheckResetException(a, "best effort");
-            long seed = random.Next();
+            long seed = random.NextInt64();
             bool useCharFilter = random.NextBoolean();
             Directory dir = null;
             RandomIndexWriter iw = null;
@@ -769,13 +764,13 @@ public void CheckRandomData(Random random, Analyzer a, int iterations, int maxWo
 #if FEATURE_INSTANCE_TESTDATA_INITIALIZATION
                     this,
 #endif
-                    new Random((int)seed), dir, a);
+                    new J2N.Randomizer(seed), dir, a);
             }
 
             bool success = false;
             try
             {
-                CheckRandomData(new Random((int)seed), a, iterations, maxWordLength, useCharFilter, simple, offsetsAreCorrect, iw);
+                CheckRandomData(new J2N.Randomizer(seed), a, iterations, maxWordLength, useCharFilter, simple, offsetsAreCorrect, iw);
                 // now test with multiple threads: note we do the EXACT same thing we did before in each thread,
                 // so this should only really fail from another thread if its an actual thread problem
                 int numThreads = TestUtil.NextInt32(random, 2, 4);
@@ -785,7 +780,7 @@ public void CheckRandomData(Random random, Analyzer a, int iterations, int maxWo
                 {
                     threads[i] = new AnalysisThread(seed, startingGun, a, iterations, maxWordLength, useCharFilter, simple, offsetsAreCorrect, iw);
                 }
-                
+
                 foreach (AnalysisThread thread in threads)
                 {
                     thread.Start();
@@ -948,7 +943,7 @@ private static void CheckRandomData(Random random, Analyzer a, int iterations, i
                     {
                         // TODO: really we should pass a random seed to
                         // checkAnalysisConsistency then print it here too:
-                        Console.Error.WriteLine("TEST FAIL: useCharFilter=" + useCharFilter + " text='" + Escape(text) + "'");
+                        Console.Error.WriteLine($"TEST FAIL (iteration {i}): useCharFilter=" + useCharFilter + " text='" + Escape(text) + "'");
                         throw; // LUCENENET: CA2200: Rethrow to preserve stack details (https://docs.microsoft.com/en-us/visualstudio/code-quality/ca2200-rethrow-to-preserve-stack-details)
                     }
                 }
@@ -1201,8 +1196,8 @@ private static void CheckAnalysisConsistency(Random random, Analyzer a, bool use
             }
             reader = new StringReader(text);
 
-            long seed = random.Next();
-            random = new Random((int)seed);
+            long seed = random.NextInt64();
+            random = new J2N.Randomizer(seed);
             if (random.Next(30) == 7)
             {
                 if (Verbose)
@@ -1248,7 +1243,7 @@ private static void CheckAnalysisConsistency(Random random, Analyzer a, bool use
             if (field != null)
             {
                 reader = new StringReader(text);
-                random = new Random((int)seed);
+                random = new J2N.Randomizer(seed);
                 if (random.Next(30) == 7)
                 {
                     if (Verbose)
@@ -1297,7 +1292,7 @@ internal static int[] ToIntArray(IList<int> list)
         /// <summary>Returns a random <see cref="AttributeFactory"/> impl</summary>
         public static AttributeFactory NewAttributeFactory(Random random)
         {
-            switch (random.nextInt(2))
+            switch (random.Next(2))
             {
                 case 0:
                     return Token.TOKEN_ATTRIBUTE_FACTORY;

src/Lucene.Net.TestFramework/Analysis/CrankyTokenFilter.cs

@@ -62,7 +62,7 @@ public override void End()
         public override void Reset()
         {
             base.Reset();
-            thingToDo = random.nextInt(100);
+            thingToDo = random.Next(100);
             if (thingToDo == 2 && random.nextBoolean())
             {
                 throw new IOException("Fake IOException from TokenStream.Reset()");

src/Lucene.Net.TestFramework/Analysis/MockAnalyzer.cs

@@ -1,6 +1,7 @@
 using Lucene.Net.Support.Threading;
 using Lucene.Net.Util;
 using Lucene.Net.Util.Automaton;
+using RandomizedTesting.Generators;
 using System;
 using System.Collections.Generic;
 using System.IO;
@@ -72,7 +73,7 @@ public MockAnalyzer(Random random, CharacterRunAutomaton runAutomaton, bool lowe
             : base(PER_FIELD_REUSE_STRATEGY)
         {
             // TODO: this should be solved in a different way; Random should not be shared (!).
-            this.random = new Random(random.Next());
+            this.random = new J2N.Randomizer(random.NextInt64());
             this.runAutomaton = runAutomaton;
             this.lowerCase = lowerCase;
             this.filter = filter;

src/Lucene.Net.TestFramework/Analysis/MockGraphTokenFilter.cs

@@ -1,5 +1,6 @@
 using Lucene.Net.Analysis.TokenAttributes;
 using Lucene.Net.Util;
+using RandomizedTesting.Generators;
 using System;
 using Console = Lucene.Net.Util.SystemConsole;
 
@@ -38,13 +39,13 @@ public sealed class MockGraphTokenFilter : LookaheadTokenFilter<LookaheadTokenFi
 
         private readonly ICharTermAttribute termAtt;
 
-        private readonly int seed; // LUCENENET specific: changed to int, since .NET random seed is int, not long
+        private readonly long seed;
         private Random random;
 
         public MockGraphTokenFilter(Random random, TokenStream input)
             : base(input)
         {
-            seed = random.Next();
+            seed = random.NextInt64();
             termAtt = AddAttribute<ICharTermAttribute>();
         }
 
@@ -111,7 +112,7 @@ public override void Reset()
             // NOTE: must be "deterministically random" because
             // baseTokenStreamTestCase pulls tokens twice on the
             // same input and asserts they are the same:
-            this.random = new Random(seed);
+            this.random = new J2N.Randomizer(seed);
         }
 
         protected override void Dispose(bool disposing)