cmake: Use plain CMake instead of Zephyr's custom interface

[tejlmand]

Zephyr are having their own wrapped versions of many CMake commands.
The commands looks similar, many of them are simply re-arranging words you need
to type.

Some drawbacks I see with the current approach are:

• Additional maintenance, both the commands themselves and the documentation
• Additional set of API. A user of Zephyr cannot just learn CMake commands and
  use CMake's online documentation as reference, the user also has to read/learn
  Zephyr additional set of API's.
• Slight re-arranging of words is not hiding complexity, it is just re-arranging
  it
  e.g. zephyr_sources(...) => target_sources(zephyr ...)
  The Zephyr just moves from being target name, to be the function name instead
• Risk associated with the use of macros (scoping)
• Hiding the library name is similar to hide a variable name in a programming
  language.

In CMake you will do:
add_library(MyLib ...)
target_sources(MyLib ....)
target_include_directories(MyLib ...)

This is with slight variation similar to an OO language, where one might have done:
MyLib = new library()
MyLib.sources(...)
MyLib.include_directories(...)
The object name in CMake is just placed as argument.

but in Zephyr, the object/variable name has disappeared:
zephyr_library()
zephyr_library_sources(...)
zephyr_library_include_directories(...)

Example of functions with the words re-arranged,
Instead of writing target_sources(zephyr PRIVATE ...), I have to write
zephyr_sources(), so target name zephyr is just part of the function name.

Instead of target_sources( PRIVATE ...), the target name is hidden and I
need to write zephyr_library_sources(...).

I still need to know the difference between zephyr_sources and
zephyr_library_sources, where CMake just has a single API.

One might argue, that you avoid to know the difference between PRIVATE, PUBLIC, and INTERFACE.
But those keywords are easy and useful to know:
PRIVATE : The content following is only used by the library.
PUBLIC : The content is used by the library and those who links to it.
INTERFACE: The content is only used by those who links to the library.

Zephyr's wrapped commands doesn't completely remove those keywords, it just
places them elsewhere, e.g.
zephyr_library_named() = add_library( STATIC "")
zephyr_interface_library_named() = add_library( INTERFACE)

The use of macros must be done with care:
As example, a user cannot use the variable: name,
as this name is used by the Zephyr macros,
Example, do the following in any CMakeLists.txt, and you'll see the macro
manipulates ${name}.
(CMake functions doesn't suffer from this problem):
set(name "MyName")
message("Value of name=${name}") # The value of name=MyName
zephyr_library()
message("Value of name=${name}") # Now the value of name is something else

I acknowledge that some commands are providing an easier out-of-the box
understanding by providing _ifdef versions, but I believe this can also be
achieved without the drawbacks mentioned above.

List of Zephyr commands that wraps CMake commands and their CMake equivalent:
zephyr_sources = target_sources(zephyr PRIVATE)
zephyr_include_directories = target_include_directories(zephyr_interface INTERFACE)
zephyr_system_include_directories = target_include_directories(zephyr_interface SYSTEM INTERFACE)
zephyr_compile_definitions = target_compile_definitions(zephyr_interface INTERFACE)
zephyr_compile_options = target_compile_options(zephyr_interface INTERFACE)
zephyr_link_libraries = target_link_libraries(zephyr_interface INTERFACE)

zephyr_library() = add_library( STATIC "") # With a library name based upon the folder
zephyr_library_named() = add_library( STATIC "")
zephyr_link_interface() = target_link_libraries( INTERFACE zephyr_interface)
zephyr_library_sources() = target_sources( PRIVATE )
zephyr_library_include_directories(

) = target_include_directories( PRIVATE )
zephyr_library_link_libraries() = target_link_libraries( )
zephyr_library_compile_definitions() = target_compile_definitions( PRIVATE )
zephyr_library_compile_options() = target_compile_options( INTERFACE )
zephyr_interface_library_named() = add_library( INTERFACE)

EDIT: This is part of umbrella issue: #8827

[tejlmand]

Issue #8614 further shows that wrapping of CMake commands limits the feature set of CMake.
And often when such issues arises, then additional wrapped commands gets introduced

[tejlmand]

Additionally, another issue with wrapped function, is that the behavior of the wrapped functions are not stable as their internals can change between commits.
Thus the behavior which were expected at the time of using the zephyr API, might unexpectedly change later.

This is seen in the suggested fix, referred to by this comment:
#8614 (comment)

The CMake API is in contrast very stable, and behave consistently.

Also, wrapping if CMake API this way, limits the feature set of CMake's API, and thus, each time such blocked feature is needed, a new zephyr wrapper gets introduced.

[SebastianBoe]

I'm not sure what to make of this, could you change the title to make it clear whether this is a;

feature request,
question,
bugreport,
RFC.

[tejlmand]

Consider it a "reverse" feature request.

The request is to remove the mentioned wrapped functions, and instead stick to default CMake functions, which already provides what is needed.

Only create and maintain zephyr functions where a new functionality is really useful, e.g. like the zephyr check compiler flag functionality.

This will also ease the entry barrier for new comers who already know the CMake language.

By using plain cmake for target handling we avoid having to create new wrapper functions whenever a new CMake keyword / function argument needs to be supported, such as:
#8614

And also we avoid e.g. PR's as:
#9309

There are probably more out there.
See this as part of: #8827

[meuter]

As a newcomer to zephyr, I can attest that the whole wrapper macro thing is very frustrating.
CMake is arguably difficult to master, but once you do, it gives you a lot of flexibility. Having to learn an extra domain specific language on top of that feels redundant IMHO. I would suggest to stick with vanilla cmake and only supplement where it make sense.

[mped-oticon]

@meuter I agree, but CMake adheres to an opinionated I-will-deduce-everything-for-you mentality that collides with the nitty-gritty specifics required to build an OS. So I see the reason for why things are the way they are. Add to that, the need to support multiple wildly-different toolchains...

If you see improvements, feel free to send a Request-For-Comments pull request.

[meuter]

@mped-oticon I can very well understand the kind of details and complexity you're talking about and indeed it's bit of a culture clash. However, I think a more "transparent" approach would be easier to grasp. Currently the zephyr_xxx() and zephyr_library_xxx() macros are just trying to hide some of the complexity and are not really adding much to what cmake offers out of the box.

Another advantage I see in keeping with the more idiomatic cmake way of doing things is that it would make porting existing libraries to zephyr much easier (most are coming with CMakeLists.txt these days). You'd simply add_subdirectory(/path/to/external/library) and you should be mostly good to go.

In any case, once I have a bit more experience with zephyr (as I said I'm a newcomer), as you suggest I'll send a Request-For-Comments pull request.

[mbolivar-nordic]

@tejlmand I am pretty sure this can be closed with resolution "won't do"; right?

[tejlmand]

I wouldn't say so, but it's down on the prioritization list compared to other work.

Also I think it's nice to keep open for other new comers to Zephyr who might be experienced CMake developers that this is something we are aware about.

[mbolivar-nordic]

Hah, okay then. I'm officially taking bets on if this ever gets done :)

[boaks]

Very interesting discussion.

Is there an idea, how that is handled for "third-party/external libraries"?
I feel a little strange about that and it would be great, if there is a link to something as best practice.
I will create also a separate issue. Adding a comment here is done in hope to reach the experts more directly.

[zephyrbot]

Hi @tejlmand,

This issue, marked as an Enhancement, was opened a while ago and did not get any traction. Please confirm the issue is correctly assigned and re-assign it otherwise.

Please take a moment to review if the issue is still relevant to the project. If it is, please provide feedback and direction on how to move forward. If it is not, has already been addressed, is a duplicate, or is no longer relevant, please close it with a short comment explaining the reason.

Thanks!

[clamattia]

What is the status of this?
Thanks

[canatella]

Btw, not using a CMAKE_TOOLCHAIN_FILE is also part of not using plain CMake.

This would still be a nice to have regarding integration with Conan (see #31745) and probably other projects.

I resorted to copy paste the compile flags used by Zephyr in a Conan profile where I could just have pointed to the toolchain file if it was there. Then I discovered that find_zephyr does it self end up calling project which created some recursive error with cmake if I included it in any subproject.

I think I understand where you're coming from, but when we recently started our new project, we thought "cool, Zephyr is CMake based, it'll be easy to integrate with our current code". Then I was disappointed to see it's yet another project working around CMake in its own unique, incompatibles ways. Reading the few issues around this here makes it clear there are actual reasons about this, and I understand that there's a lot of cmake code that would need to evolve, but it would be nice if there were some plan to move toward a more standard way of doing things.

At least providing CMAKE_TOOLCHAIN_FILE support to enable compilation in other projects. No need for flashing or linking, but at the very least this would provide an easy way to build a static library to be integrated in the project later, without having to make it a Zephyr specific project so that if you're using conan, vcpkg or any other wrapper you can just drop -DCMAKE_TOOLCHAIN_FILE=zephyr/my-board.cmake and be good to go.

The KConfig configuration system could be replaced by cmake variables. ccmake and cmake-gui can be used to configure the variables. Also with presets supports, it's easier now to share common configuration settings.

And then maybe merging some features upstream? The global compiler flag cache is a nice thing and could maybe be useful to other projects. For the compiler features, maybe using try_run and try_compile with cmake -C common-cache.txt to populate the cache upfront could be a more standard way to do that? There's now preset files which could also be used to specify board specific variables.

Just my 2 cents here. All in all, it's working. I'm just sad that in 2024, it's still complicated to integrate two unrelated C++ projects even though they both use CMake.

Best wishes for 2025 ;)

[TorstenRobitzki]

My main problem with the Zephyr approach of using CMake is, that it prohibits a lot of features in CMake. In a lot of projects I use CMake and I havn't seen so far anything special, that would justify the "weird" use of CMake that I'm seeing in Zephyr (but of cause, I don't know all the internals).

Beside the home grown compiler detection (which then requires the weird find_package before project), defining the OS itself as the one and only executable target is very limiting:

• In larger projects, it is usual to group source into libraries, define dependencies between that libraries and then define executable targets, that depend on different sets of libraries. That makes it possible to create multiple executables that are all suitable for the very same hardware. For example: Special Device Driver tests, Factory Acceptance Test binaries, the application binary itself.
• The need, to either wrap every binary into its own Zephyr project, largely increases build times. Or alternatively, define the set of files to compile with build configurations, makes the build way more complicated.
• There are projects, that make use of multiple microcontrollers. With zephyr, I would have to build the firmware for every microcontroller within it's own Zephyr project, which adds the need to some more external scripting to orchestrate all builds (CMake has this already out of the box).

There seems to be even more redundant features being implemented in Zephyr, that already exists in CMake (where it is usually very well documented):

• sysbuild: The ExternalProject package can be used to orchestrate the build of different binaries. (Even the package manager part of west could be easily implemented in CMake)
• VERSION: That's a feature that's already build into CMake.

I wonder, why Zephyr is not just used as a library (maybe having a function to configure and create an object library to be used). Something along the lines of:

cmake_minimum_required(VERSION 3.23)

project(I_Have_A_Dream VERSION 1.0.0 LANGUAGES C CXX)

find_package(Zephyr REQUIRED)

create_zephyr_os(
    LIBRARY_TARGET_NAME vanila
    KCONFIG prj.config
    HARDWARE "nrf5340dk/nrf5340/cpuapp"
    OVERLAY did_und_dat.overlay)

add_executable(firmware main.cpp)
add_library(firmware PRIVATE vanila)