Split and group tests to run subsets in parallel

[doron-cohen]

I would like to suggest a splitting feature to group tests to subsets and run them separately.

There is this tool: https://github.com/Songmu/gotesplit but I think it will be hard to make it work well with gotestsum.
I also tried working with go test ./... -list=. which should list all tests but the issue is that it runs TestMain and executes our init code for integration tests.

I really wish I could use gotestsum like gotesplit.

Something like:

# Run the first subset of three
gotestsum --split-subsets=3 --split-index=1

Or:

# Run the 3rd subset of five packages (with tests)
gotestsum --split-packages=5 --split-index=3

These could come in really handy to split integration tests to several jobs in the CI and thus shorten the dev cycle.
Thanks.

[dnephin]

Thanks for opening this issue! I believe #277 may do what you want. That branch still needs some work, primarily some new tests. This new gotestsum feature is a little smarter than gotesplit because it uses previous run time of tests to find an optimal splits. Unfortunately it still does require running go test ./... -list=. I don't think there's any other way to get the list of test names to run. If you were to use the list of test names from the previous run you'd never run new tests.

Maybe you can change TestMain to not run in some cases, or have the first test to run do the initialization?

Option 1

If go test -list works without the logic in TestMain (I expect it should in most cases), I think you could skip it with something like this. I believe the flag is -test.list when it gets received by the test binary, but I haven't tested it.

func TestMain(m *testing.M) {
    if !isTestList() {
    // do init code here
    ....    
    }
    os.Exit(m.Run())
}

func isTestList() bool {
    for _,  arg := range os.Args {
        if arg == "-test.list" || strings.HasPrefix(arg, "-test.list") {
            return true
        }
    }
    return false
}

Option 2

If you have a common setup function for all the tests, you could move the initialization into a function that isn't TestMain and use sync.Once to call it. That way your initialization is not run when listing tests.

var initOnce sync.Once

func setup(t *testing.T) {
    // do shared setup first
    initOnce.Do(func() {
        // init logic that used to be in TestMain
        ...
    })
    ...
}

I hope that helps!

[doron-cohen]

That is very helpful @dnephin thanks!
I just don't get the new --partition-tests-in-package flag. Will it run group tests from different packages or will it run groups of test packages?

[dnephin]

Both should be possible.

The original feature in #258 splits a test run by package, so each split is a group of packages.
The flag --partition-tests-in-package=<pkg> would be to change the behaviour to split the tests in a single package into groups.

[doron-cohen]

This feature will greatly help us. Is there anything I can do to help with merging it?

[dnephin]

Thanks for having a look! I think the main thing blocking a merge is some more test coverage.

It would be great to understand how you're planning on using this. Would you split by package, or split the tests in one package? Do you use github actions, or would you be looking to use this in a different CI system?

I think one of the limitations right now is that we have to run a job first to go test -list, which can add some time to the CI run. You can see what that looks like here: https://github.com/dnephin/infra/actions/runs/3451647679

That first job can easily add 30s+ to the total runtime. It depends on how long it takes to build the project to list the tests. That extra time is probably saved by splitting the test run into parallel batches, but for smaller test suites it might reduce the time saved a bit.

[doron-cohen]

We would like to run integration tests in batches. But, we want to run all tests in a certain package together since it shares an initialization code (spinning up test-containers) we want to only run once. All of this runs on Github Actions.

I didn't test how long will go test -list take once we use the trick in your comment to avoid running TestMain when listing. This is something we will need to test.