Allow running just unit or integration tests in cargo test

[jplatte]

Problem

cargo test has a flag for running just doctests (--doc) but equivalent flags for running just unit tests (compiling the crate with cfg(test) but not compiling integration tests) or integration tests (compiling the crate without cfg(test) but compiling and running integration tests don't seem to be available.

This would be useful for improving CI times for crates with many tests, as you could run these two things in separate CI jobs and parallelize them that way.

Proposed Solution

Add --unit-tests and --integration-tests flags, or some equivalent.

[epage]

equivalent flags for running just unit tests

There is:

--lib                        Test only this package's library unit tests

integration tests

There is this though it requires some more work to get the behavior you are looking for

--test <NAME>...             Test only the specified test target

Though if you switch to one or only a couple of test targets like matklad talks about, it will improve link times in your CI and make it easier to reference your black box tests. toml-rs took the approach of splitting the black box tests into a separate crate which allows specializing the dependencies which would also improve your unit test CI times.

This would be useful for improving CI times for crates with many tests, as you could run these two things in separate CI jobs and parallelize them that way.

imo Splitting on just unit vs integration test is only a stop gap measure for helping with the problem of CI times. A lot of smaller crates won't be helped because test times will be so small that CI jobs will all be compilation time and by splitting it up into multiple jobs, users will just hit their max concurrent runner cap, serializing their CI jobs and slowing down their overall CI.

For users with larger test sets and "unlimited" number of concurrent runners, the solutions I tend to see outside of Rust that would be good starting points to look to are:

• Crate-level test avoidance which is being experimented on in cargo-guppy. This will help both in reducing the number of tests but also compilation time
• Case-level test avoidance driven by past coverage reports
• Test sharding

[jplatte]

equivalent flags for running just unit tests

There is:

--lib                        Test only this package's library unit tests

What if I have a binary with unit tests?

---

Re. the rest of your response, I see what you mean and I'll think about applying some of these things to the projects I work on.

I still feel like having these sorts of flags could be nice though – in particular in workspace crates where test output is often long, it would be beneficial to go over smaller sets of tests locally and re-run just those (with fewer stuff to rebuild) when something fails. If that doesn't convince you though, feel free to just close this issue. If it seems unlikely this would be implemented I can live with that.

[epage]

What if I have a binary with unit tests?
Those are broken out in

--bins                       Test all binaries--bins                       Test all binaries

[weihanglo]

What if I have a binary with unit tests?

cargo test --bin <bin> or cargo test --bins

---

I should state that internally Cargo doesn't have the concept of unit/integration tests. All it does is to invoke rustc with extra arg (e.g. --test), and then libtest harness takes over to build a ad-hoc binary that runs those fn with #[test] ¹. The major difference between so-called "unit tests" and "integration tests" is the ability to access non-pub items in your library target. This paragraph² gets a decent explanation IMO.

Since you've mentioned doctests, the behaviour is quite different from other kinds of tests. Here is a related comment³ from me you might be interested in. The Cargo book haven't yet covered it well, but yeah, doc improvements are always welcome!

BTW, it could help more if you can provide a concrete example for your use case.

Footnotes

1. https://doc.rust-lang.org/1.59.0/rustc/tests/index.html ↩

2. https://doc.rust-lang.org/1.59.0/cargo/reference/cargo-targets.html#tests ↩

3. https://github.com/rust-lang/cargo/issues/10361#issuecomment-1043872481 ↩

[ehuss]

Just to be clear, this can be done today:

• Unit tests: cargo test --lib --bins
• Integration tests: cargo test --test *

[jplatte]

I guess there's really no point in keeping this open then. There seem to be some related improvements to be had, but I'm no longer convinced having more (dedicated) flags for these things would be an improvement.