Spec: Add ability to randomize specs

[Fryguy]

This PR adds the ability to run specs in a random order. To do so, just pass --order random to crystal spec. Randomized specs will output a seed which can be used if the tests need to be run in the same previous random order. To do so, just pass --order <seed-value>.

Default

With random

With seed

---

Note that this PR takes some of the test infrastructure stuff I built in #8242 in order to test the randomization.

[Fryguy]

@asterite Updated.

[bew]

Does it randomize things level per level? Or is it global?

Given 2 describes A/B/C, with 3 it each, a normal run would do:

A.1
A.2
A.3
B.1
B.2
B.3
C.1
C.2
C.3

With your PR, will it randomize like:

B.1
B.3
B.2
C.2
C.1
C.3
A.1
A.3
A.2

Or like:

B.3
C.2
B.2
A.1
A.3
C.1
B.1
C.3
A.2

?

[asterite]

@bew I would expect randomization to happen inside individual describe/context blocks, because doing it globally is basically impossible considering we'll have before_all hooks and such

[straight-shoota]

Comparing with rspec, it has two different options:

--order TYPE[:SEED]            Run examples by the specified order type.                            
                                 [defined] examples and groups are run in the order they are defined
                                 [rand]    randomize the order of groups and examples               
                                 [random]  alias for rand                                           
                                 [random:SEED] e.g. --order random:123                              
--seed SEED                    Equivalent of --order rand:SEED.                                     

I suppose it's fine to merge them into one since there don't seem to be any additional use cases.

rspec's default seed value is in 0 .. 2 ** 16, which means a decimal number with at most 5 digits. This is much easier to handle than a 16 digit hexadecimal seed. And these smaller seed values should be totally sufficient.

[Fryguy]

Does it randomize things level per level? Or is it global?

Per-level. This is particularly important for verbose output and before/after hooks, otherwise it would interleave improperly.

Comparing with rspec

I find rspec's CLI confusing, and this seemed a lot simpler. I can change though if we all agree on a different CLI interface.

rspec's default seed value is in 0 .. 2 ** 16, which means a decimal number with at most 5 digits. This is much easier to handle than a 16 digit hexadecimal seed. And these smaller seed values should be totally sufficient.

When I started this exercise I was pulling the values out of the Random::PCG32, which was a pair of UInt64s, and I converted those to a hex string for ease of use. But now, after @asterite's suggestion to generate a seed right in the Spec module, I can definitely do this, since I control the value.

[Fryguy]

Updated with all suggestions. (EDIT: Also updated the OP with new screenshots)

For the random seed range I chose 1..99999 mainly because it's simpler to ensure 5 or less digits with a basic regex that have to try to convert it and ensure it's 1..2^16. As the number is arbitrary anyway, I figured this was good.

[oprypin]

Hmm I don't like the strings being used here. At least it could be symbols. Or change it to seed=, random_seed.

[Fryguy]

The methods in this DSL module are basically the implementation of the CLI, so I'd like to keep it as close to the CLI as possible. This includes incoming values as strings, and the method being named order= to match the --order CLI param.

[straight-shoota]

I have comments about a few minor details, but this is (almost) good to go!

[Fryguy]

@straight-shoota @RX14 I've rebased this and fixed the small comments, so this should be good to go.