How should we disable building tests for CLI11 when added as subproject?

[kcsaul]

Hi,

Having found/added CLI11 to my project previously (very pleased to find it!), I've recently upgraded to the latest version.

Rather than copying files or using more complicated commands for including external projects, I prefer to add external projects as a git submodule with cmake add_subdirectory to include them in the overall project build.

After upgrading to the latest version, I'm now getting build failures because the googletest library has already been defined when googletest was included by CLI11 (before being included directly for my project tests).

It would seem this has come about due to #183 forcing the CLI11_TESTING option based on the global BUILD_TESTING option which is defined when including CTest (something I do before including external projects so option can be used to suppress including googletest in the build).

Based on comments in a similar issue for a different project, it would seem the only way of disabling tests for a subproject is it uses a specific option (like CLI11_TESTING), as there's no way of overriding/presetting the global BUILD_TESTING option for a subproject, without disabling tests for main project.

I note there's also comments in #183 which indicate CLI11_TESTING will be deprecated/removed in a future release.

Unless there's another way of achieving it(?), I'd prefer the use of CLI11_TESTING be kept, perhaps defaulting based on BUILD_TESTING (if not already preset before including CLI11).

Thanks,
Kevin

[henryiii]

I've been working on ideas related to this. Because BUILD_TESTING is a global value, it clashes easily. I think I had a solution: only add testing if this is the main project and BUILD_TESTING is defined, or if CLI11_TESTING is defined. You can see the discussion here: https://gitlab.com/CLIUtils/modern-cmake/issues/7 .

This need to be addressed before the 1.8 release.

[phlptp]

I have found it is handy in subprojects and libraries that are used as components to have some ability to turn off testing even if they are not directly a subproject. So I would recommend having CLI11_TESTING be checked as well if it is defined. So setting CLI11_TESTING to OFF would turn off the build of all the testing regardless of other variables and project scope.

[kcsaul]

Here's a variation on the initial solution to consider that may be a little more flexible for those who want to explicitly enable/disable CLI11 testing.

• Default CLI11_TESTING based on whether it's the main project, provided it's not been defined/cached already.

• Before enabling/including tests, check both CLI11_TESTING and BUILD_TESTING (if defined).

[henryiii]

Slightly different solution; most of the time you probably do not want all subprojects to build their test suites. So:

Main Project	CLI11_TESTING unset	CLI11_TESTING OFF	CLI11_TESTING ON
BUILD_TESTING unset	✔️	❌	✔️
BUILD_TESTING OFF	❌	❌	❌
BUILD_TESTING ON	✔️	✔️	✔️

Sub Project	CLI11_TESTING unset	CLI11_TESTING OFF	CLI11_TESTING ON
BUILD_TESTING unset	❌	❌	✔️
BUILD_TESTING OFF	❌	❌	✔️
BUILD_TESTING ON	❌	❌	✔️

[phlptp]

The one I would find most confusing here is BUILD_TESTING=ON and CLI11_TESTING=OFF then the tests still build as a main project. I think there should be a single option that when turned off explicitly guarantees the tests are turned off.

[henryiii]

This one's the hardest - but most of the time, you would not define both in a main project. If you do, then BUILD_TESTING is the "standard" option, so it takes priority over CLI11_TESTING. But I believe I could reverse this. Or I could take an AND or an OR, I think.

I could also drop CLI11_TESTING and never allow tests to be built as a subproject - not sure how well it really supports this anyway nor why one should do it.

[henryiii]

New version:

Main Project	CLI11_TESTING unset	CLI11_TESTING OFF	CLI11_TESTING ON
BUILD_TESTING default	✔️	❌	✔️
BUILD_TESTING OFF	❌	❌	✔️
BUILD_TESTING ON	✔️	❌	✔️

Sub Project	CLI11_TESTING unset	CLI11_TESTING OFF	CLI11_TESTING ON
BUILD_TESTING unset	❌	❌	✔️
BUILD_TESTING OFF	❌	❌	✔️
BUILD_TESTING ON	❌	❌	✔️

[metopa]

I think the last approach is very reasonable. Giving CLI11_TESTING a priority makes a lot of sense.

[henryiii]

I like it - it also hides CLI11_TESTING (it's not predefined, so it's not in cmake -L), so main-project builds sees BUILD_TESTING and non-main projects really shouldn't be turning on subproject tests. That's not why add_subdirectory should be used here - you woudn't expect to be able to build the tests for a find_package package. But it's there if someone really wants it.

[henryiii]

But you'd expect someone who knows about CLI11_TESTING and defines it to know what they want.