[CMake] Support per-target linker flags

[thevinster]

CMAKE_{C/CXX}_FLAGS affects all targets in LLVM. This can
be undesirable in situations, like the case of enabling thinLTO,
where -flto is added to every source file. In reality, we only
care about optimizing a select few of binaries, such as clang or lld,
that dominate the compilation pipeline. Auxiliary binaries in a
distribution and not on the critical path can be kept non-optimized.
This PR adds support of per-target linker flags, which can solve the
thinLTO problem by negating the effects of LTO via targeted linker
flags on the targets. The example of negating thinLTO
above can be done by doing the following:

set(LLVM_llvm-dwarfdump_LINKER_FLAGS "-Wl,--lto-O0" CACHE STRING "Custom linker flags to llvm-dwarfdump")
set(LLVM_lldb_LINKER_FLAGS "-Wl,--lto-O0" CACHE STRING "Custom linker flags to lldb")

There's other applications where this could be used (e.g. avoid
optimizing host tools for build speed improvement etc.).
I've generalized this so that users can apply their desired flags to
targets that are generated by llvm_add_library or add_llvm_executable.

Internally, our toolchain builds were on average 1.4x faster when
selectively choosing the binaries that we want optimized.

[MaskRay]

I have wanted this for a long time. Thank you for the patch. (I am not familiar with CMake.) Does this variable need to be cached? New CMake variables can be documented at llvm/docs/CMake.rst

Internally, our toolchain builds were on average 1.4x faster when selectively choosing the binaries that we want optimized.

1.4x as fast as the baseline or the new speed is 240% of the baseline?

[smeenai]

Looks useful :) Do we need the granularity of the exe vs. module vs. shared custom flags, or could we just have a single set of custom flags per target? (That'd also allow you to have a common function that you just call from both llvm_add_executable and llvm_add_library instead of duplicating any logic.)

[thevinster]

Does this variable need to be cached?

Yes, otherwise LLVM will not pick up these values. I'll add it to the documentation. Thanks for the pointer.

1.4x as fast as the baseline or the new speed is 240% of the baseline?

1.4x as fast as the baseline. So, as an example, if an overall build took 1 hour to build everything with LTO, it would've taken us around ~43 minutes to build our toolchains with LTO because we selectively only chose what we wanted to optimize. YMMV depending on the machine instance and what you decide to not optimize.

[thevinster]

Do we need the granularity of the exe vs. module vs. shared custom flags, or could we just have a single set of custom flags per target?

Initially, I separated because I thought the flags would've been duplicated, but I now realize that the two methods are mutually exclusive (i.e. clang target would not be invoked with llvm_add_library and same goes for a library being invoked with llvm_add_executable)

[thevinster]

@petrhosek Does this change look reasonable to you? I know this particular issue could be solved with -ffat-lto-objects, but I think this could apply to other places as well.

[nikic]

What is the purpose of LLVM_CUSTOM_BINARIES here? Why is just LLVM_<target>_LINKER_FLAGS not sufficient?

[thevinster]

Hmm, you're right. It isn't providing any use here. I had originally thought of having it so we can could write specific logic against targets that have been added with custom flags, but I realize that might not be a real use case after all.

[petrhosek]

I'd omit this to avoid the extra build spam (or make it optional).

[thevinster]

What did you mean by optional here? Like a CMake flag that enables the message? In general, I find this message to be quite helpful when debug whether targets were applied with custom flags without needing to go into the CMakeCache.txt.

[thevinster]

I'm going to land this as is, and if this becomes very build spammy, we can figure out other approaches to disable it.

[petrhosek]

An alternative to an option would be to use the DEBUG mode instead of STATUS so developers can use CMAKE_MESSAGE_LOG_LEVEL to suppress the output if desired.

[thevinster]

#69497

[petrhosek]

Reducing the overhead of LTO was one of the motivations for `-ffat-lto-objects, which let you use LTO for distribution components while disabling LTO for everything else; @ilovepi is working on a patch for that which should be ready for review soon.

This change might be still useful if you want to enable/disable LTO only for certain targets. My concern is the granularity, specifically how practical is controlling linker flags at the per-target level, especially if you build a large number of targets? The alternatives I could think of is setting the linker flags at the per-distribution granularity.

[thevinster]

The alternatives I could think of is setting the linker flags at the per-distribution granularity.

For us, we define a set of distribution as well for other teams to use. But even with our filtered down distribution, we'd still like to only apply -Wl,--lto-O0 since tools like llvm-objdump or llvm-nm are necessary for integration with our build system for debugging, but not necessary to be LTO-ed since they aren't in the critical path of release builds.