Exhaustive tests?

[sffc]

In ICU, we have an exhaustive test mode for tests that are particularly slow. Examples of tests that might be exhaustive tests:

1. Test a feature over all locales*
2. Test the bounds of something (e.g., longest supportive string)

It would be nice if we had an exhaustive test mode in ICU4X, such that running cargo test remains fast and doesn't get stuck on slow tests.

* It's common in ICU to unconditionally test a random subset of locales, and test the full set in exhaustive mode.

[sffc]

We could add a feature like "exhaustive" and conditionally compile the tests only when that feature is enabled. It could be disabled by default.

@Manishearth @zbraniecki @hsivonen Is there a standard practice for this in Rust?

[Manishearth]

Yeah it's done sometimes

[hsivonen]

I don't have experience with practices for this.

[sffc]

Discussion 2020-10-30:

3 types of tests:

• Overview
  • Run every time you change code (e.g. in local development)
  • Total Duration: a few seconds
• Exhaustive
  • Run before checking in a feature (e.g. on every PR)
  • Total Duration: a few minutes
• Slow
  • Run occasionally (e.g. on the main branch)
  • Total Duration: up to an hour

[sffc]

I'm going to consider this resolved by #376. I am adding a new cargo quick command that runs only the fastest tests; it finishes in less than 2 seconds on my laptop for all of ICU4X. If and when we do eventually need to blacklist slow tests, we can do so in the framework proposed in #376.