west.yml: lazy module loading

[cfriedt]

Introduction

This topic should really be of interest for anyone who uses west to pull in Zephyr components, which is likely a large majority at this time.

Generally though, the topic should be of interest to end users, TSC members, maintainers, product designers, module authors, etc.

Problem description

Modules are the primary vehicle used to scale Zephyr.

At the beginning, it was fairly inconsequential that there were 5-10 modules pulling in maybe ~100 MB of additional repositories, HALs, libraries, etc. However, as Zephyr and the Zephyr Community have grown, it's become a more frequent concern that the default behaviour (west update) pulls in several GB of repository data. Moreover, for the a common goal like running hello_world on a particular board, the vast majority of modules, and the GB of git repository data downloaded, are largely unused.

There was previously an attempt to push the problem onto users by grouping west repositories into categories (core, hal, etc). However, that does not seem to have worked as well we had hoped.

Proposed change

In an ideal world, everything would be optional. Our default should be to minimize the required packages to install and the required modules to download.

We propose something along the lines of "lazy module loading". I.e. automatically updating only the the git repositories necessary for a particular app using a particular configuration running on a particular board.

Automatically downloading dependencies during a build phase are generally not a great idea, so this should be an optional feature that is disabled by default.

Some potentially "nice" features to have would be

• maybe prompt users to download repos (if in an interactive shell)?
• maybe something like a "--fetch-only" option that does the fetching in a separate step for a given app / board / config
• maybe something like a "--dry-run" print out the list of repos that would need to be downloaded, so that people can manually do that in advance for their particular CI systems
• maybe something like a "deps --list" option

Additional notes

• It's not just west modules, but the tools (e.g. ubuntu / python packages) needed to deal with those modules
  • augment module yaml schema with necessary packages
    • ubuntu: (array of deb package-names..)
    • macos: (array of package-names..)
    • windows: (array of package-names..)
    • pip: (array of package-names..)
    • redhat: (array of package-names..)
  • each of the above would of course be optional, but every module would be required to list its own dependencies outside of the most basic requirements for building Zephyr.

• West update tooling dependency management #55056

Detailed RFC

TODO

Proposed change (Detailed)

TODO

Dependencies

TODO

Concerns and Unresolved Questions

TODO

Alternatives

Deja-vu - we have had discussions and solutions about this several times.

[cfriedt]

Some PRs with associated conversations about this topic

• thrift: module integration, samples, and tests #54013
• Introduce Pigweed Module #53760

[galak]

I think one question is what kinda of means do we convey how a given module is needed. For example, is that via some Kconfig symbols?

[cfriedt]

Also, just throwing this out there, but we could consider this as a problem for a GSoC contributor 🤷

[mgielda]

Thanks @cfriedt for opening up the issue. I think it's probably more important to us than a GSoC proposal, which after all 1) will not necessarily be picked and 2) can fail (not that we can't have a GSoC about it, but then probably we should wait until the GSoC completes with undertaking other attempts; I assume the ideal GSoC project should not be on a critical path, just to avoid unnecessary pressure on the student).

I am for an apt-like behaviour w/r to asking the user - i.e. by default it will stop on recomending packages to download and wait for approval, you have to explicitly override that behavior but then everyone assumes you know what you are doing. A --download-all flag would probably make sense too.

For determining modules, Kconfig makes sense to me too.

[cfriedt]

Yea, I just won't have time to do this myself. So cannot be the assignee unfortunately. A GSoC contributor would be great in terms of someone to offload work to. Even if a GSoC fails, the work could still be used, etc. But yea, it might not necessarily be picked.

Getting this right will not be an overnight problem though, by any means. Given that we have 55 modules right now, I would expect that part of the task would be to itemize the packages that each module depends on (pip, .deb, etc). That alone could take a considerable amount of time.

[mbolivar-nordic]

@MaureenHelm MaureenHelm assigned mbolivar-nordic 5 days ago

I welcome someone developing a package manager based on west, but I have disclaimed this assignment already elsewhere.

@cfriedt feel free to reopen and assign yourself or someone else if you're aware of a volunteer.

[cfriedt]

@mbolivar-nordic - I don't know why this was assigned to you. It was originally an RFC

[marc-hb]

I welcome someone developing a package manager based on west,

Great article
https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

[cfriedt]

Great article https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

Lol - I definitely do not want to write a package manager. YAML FTW! Maybe an additional point, for that matter, should be a flag to completely silence package recommendations, because, as the article states, there are many different scenarios, and the YAML would just be a list of recommendations.

E.g. if someone wants to build Zephyr on a Linux From Scratch system, and they have built all of the required packages, then they should be able to silence warnings.

I think we just want to list recommended packages for the somewhat fixed OS versions that Zephyr supports.

[mbolivar-ampere]

one proposal based on some recent work and discussions we had: #61505

Pardon my ignorance on West, but would it be possible to take that PR then add a west install that would allow you to install any module listed in the West manifests that's not currently installed? This way you won't have to opt into all the optionals, but would be able to pick just the ones you want. Optionals would become a bit like an a-la-cart menu. It would probably be easy to add tab complete too or at least a way to lost all the manifest items that aren't yet installed.

It wouldn't be easy in the general case. The import feature makes this entire thing nontrivial because it necessarily involves a recursive process that has network and file system impact at every step in the recursion tree. I'd like to ask that people who want to get involved in this please take the time to understand how imports work before proposing solutions.

[marc-hb]

I'd like to ask that people who want to get involved in this please take the time to understand how imports work before proposing solutions.

There's an app Github label for that:
https://github.com/zephyrproject-rtos/west/issues?q=+label%3A%22Partial+imports%22+

Unfortunately not for the faint of heart... :-(

However, as Zephyr and the Zephyr Community have grown, it's become a more frequent concern that the default behaviour (west update) pulls in several GB of repository data.

BTW west init && west update --fetch-opt=--filter=tree:0 has been much faster and rock solid in our CI:

• Consider adding config option for treeless clones (--fetch-opt=--filter=...) west#638

There is of course another Github label for this sort of optimizations too:
https://github.com/zephyrproject-rtos/west/issues?q=label%3Aperformance

[uqtsherr]

Hi,

I just made a start with migrating our project to zephyr 3.5, and found that zscilib was not available. I see that this change has been integrated into v3.5 and I assume I need to add zscilib as an optional include into our west manifest/cmakelists.

Any guide as to how I go about this?
I checked the release notes and migration guide, and west modules documentation, and couldnt see any instructions.

[fabiobaltieri]

I just made a start with migrating our project to zephyr 3.5, and found that zscilib was not available. I see that this change has been integrated into v3.5 and I assume I need to add zscilib as an optional include into our west manifest/cmakelists.

Any guide as to how I go about this? I checked the release notes and migration guide, and west modules documentation, and couldnt see any instructions.

You can enable the module again by running west config manifest.group-filter -- +optional. The change was actually #61505, not this one.

@kartben should we update the 3.5 migration guide and backport it?

[kartben]

@kartben should we update the 3.5 migration guide and backport it?

Ya I was thinking just this when reading this earlier. Frankly I am not sure how we missed it but this is very much a "required change" thing.

+1 for adding it to the migration guide and making sure this also gets backported. Luckily my guess is that many people will access the migration guide from "/latest" so the only place it will still be missing is the actual 3.5.0.

[jhedberg]

+1 for adding it to the migration guide and making sure this also gets backported. Luckily my guess is that many people will access the migration guide from "/latest" so the only place it will still be missing is the actual 3.5.0.

Instead of having the full version in the URL of version-specific documentation, would it perhaps make sense to drop the last version component and just use <major>.<minor>, i.e. instead of:
https://docs.zephyrproject.org/3.5.0/
we would have:
https://docs.zephyrproject.org/3.5/

And then the version-specific documentation for a release would always be generated based on stable branch of that release (e.g. v3.5-branch in the case of Zephyr 3.5). That way we could backport important documentation fixes, and have them published on docs.zephyrproject.org. Thoughts?

[uqtsherr]

Thanks for the quick answer.

Luckily my guess is that many people will access the migration guide from "/latest" so the only place it will still be missing is the actual 3.5.0.

just checked as a layman and I hit the latest release notes and end up on the latest migration guide. Also hit latest when I go look at the v3.4.0 guide. So in my experience this is true

[fabiobaltieri]

Opened #64619

[mrx23dot]

Noob here,
in device tree we can define what HAL to use at what given version, which I guess it fetches anyway.

So all that needs to be done is not to download everything at installation, isn't it?

[glenn-andrews]

This breaks module samples like those in samples\modules\tflite-micro

[mrx23dot]

But if I didn't select a target environments then there shouldn't be any example in the first place, otherwise it's a waste of space/bandwidth.
Ideally we would start from empty env and bring in what's needed (like in platformio).

[vulpes2]

#80782 seems to be progressing nicely, once it's merged, what else do we need to implement for HAL lazy-loading?

[marc-hb]

#80782 is convenient but I don't think it will save much bandwidth and disk space and it's not about west projects either. pip was mentioned at the bottom of the description of this issue but only as something "adding fuel to the fire" and making each dependency bigger than its git clone. But it's IMHO unrelated to "lazy west downloading"

BTW Zephyr modules usually map 1:1 to West projects but they don't have to.

what else do we need to implement for HAL lazy-loading?

I'm afraid the most blatant gap is still: how does the build system divine the user's intention and needs? .config has been mentioned and it looks indeed like very relevant data but how would code automatically and generically convert that information into a list of west projects to git clone?

Much more worrying: looking beyond Zephyr, I'm not sure this problem has ever been solved in the industry. AFAIK apt-get build-dep and similar rely on a manually defined list. Usually: a non-optimized list with everything in it.

Sometimes the dependency is as simple as CONFIG_USE_THAT_LIBRARY but in general it can be much more indirect. So maybe the only thing possible here is some sort of incomplete, "semi-automated" solution that only brings the number of some_lib.h not found errors and manual installation steps to something small.

maybe something like a "--fetch-only" option that does the fetching in a separate step for a given app / board / config

For decades, developers used to perform a somewhat annoying, TWO steps "configure, make" dance with some sort of .config information in the middle connecting the two steps. CMake did not change that. So, many projects have some configure_and_build.sh or build_from_git.sh (think source.tar releases), one-step wrapper script for convenience. As the ultimate wrapper script, west build offers a single step too.

But to leverage project configuration, this new optimization would have to increase the number of build steps again. Only the first time so it shouldn't hurt convenience much but there would still be a significant impact on the user interface complexity and documentation. So I bet this would be a "CI-first" optimization. A complete Zephyr clone is big but workstation disks are bigger and the time figuring out how to leverage the optimization would probably dwarf the time downloading everything. For sure this has been my experience with optional projects #61505: because I clone rarely from scratch, I've spent more time finding the magic west incantation to download the optional modules I needed than waiting for everything to download...

But wait... doesn't the CMake configuration step already have some dependencies? The autotools ./configure step for sure has! So, .config comes too late after all?

Apologies for being my usually skeptical self but this one looks like a really tough nut to crack in general. I bet some things can be done but "semi-automated" is probably the best that can be achieved. That could still be useful, especially for CI testing that keeps repeatedly cloning from scratch every time.

For interactive use, I think manually defined, documented and configured imports is a more traditional, proven and simpler solution. Everyone can (and should?) write their own, optimized west.yml?

[mrx23dot]

Noob here.

Regarding comment above,

what if we simply use a package manager to download the selected libraries at given version into project dir on demand,
and remove no longer specified ones?

Same way requirements.txt works in python, just into project folder.
This way everything is resolvable under project dir, selfcontained.

Compilers could go system wide, still from package manager. (maybe via apt get in background)
So installing zephyr would be a small core with only package manager.

Like my recipe is

gcc ARM v666
circular buffer v1000
dio_common v5.4
dio_stm32f1 v1
...

[marc-hb]

what if we simply use a package manager ...

Do you mean a brand new, Zephyr-specific, source package manager?

.... to download the selected libraries at given version into project dir on demand,
and remove no longer specified ones?
Same way requirements.txt works in python, just into project folder.

Toolchains aside, how different is that from writing your own, optimized west.yml file?

Like my recipe is

gcc ARM v666
...
...

This issue is very "exploratory", yet it's 100% focused on west not git cloning unnecessarily. Toolchains are a completely different (and I think: very special) type of dependency. So I think they're off-topic here. BTW there is already a zephyr sdk command if you're interested, take a look at b68071d.

[mrx23dot]

When west clones out a module
dio_common v5.4
I don't think it has to be a fully linked git submodule (which easily becomes messy, and I don't care about its history),
it could just create a subdir dio_common , with some metafile/makefile saying this is v5.4 from XY place.

Cloning can happen in the background with --depth 1 into temp folder, and result copied over, so git won't be linked.
If github is currently not organised as subprojects, then it's simple file downloads from github with special version url.

It would be up to the user to version track their selfcontained project. (maybe not even with git, like with automotive MKS)

I don't think it's different from STM32 providing HAL files at given version, I copy those into my project at given checkpoint, and I don't care about their version history.

[marc-hb]

Cloning can happen in the background with --depth 1

-n --depth 1 is already passed to west in most CIs and there are other optimizations available, see links in previous comments. This issue is about the "next step" = not git cloning at all.

I don't think it has to be a fully linked git submodule project (which easily becomes messy, and I don't care about its history),

That's an optimization idea but it's a different one.

into temp folder, and result copied over, so git won't be linked.

Do you mean: deleting git clones? This would save a bit of disk space - but not that much because .git/ is compressed and often smaller than the checkout; especially when cloning with -n -o=--depth=1. I just tested that with Zephyr release v4.0.0 and deleting all modules/**/.git directories saves only 1.1G out of 4.5G total (3.5G modules/** total). Deleting .git information also neuters west status/diff/compare which means accidental changes will be missed (much worse than "messy") and it may break some build scripts (e.g.: git describe)

Deleting .git/ directories does of course not save any download time.

[marc-hb]

Much more worrying: looking beyond Zephyr, I'm not sure this problem has ever been solved in the industry. AFAIK apt-get build-dep and similar rely on a manually defined list. Usually: a non-optimized list with everything in it.

There are solutions to manage C/C++ dependencies but unlike most other languages, these solutions are not part of the C/C++ standard. To be fair, most other languages don't stretch as far and wide as C/C++ either and rarely care about embedded environments supported by Zephyr. For instance, https://learn.microsoft.com/en-us/vcpkg/get_started/get-started-packaging supports "only" Windows, macOS and Linux.

These solutions tend to come with their own build infrastructure that would clash with some or all of what west and the Zephyr build system already do.

Interestingly, most solutions seem to start with a... manifest.
https://docs.conan.io/2/tutorial/consuming_packages/build_simple_cmake_project.html
https://moderncppdevops.com/pkg-mngr-roundup/
I'm afraid all these manually defined manifests demonstrate that automatically guessing what the developer wants/needs is a really hard problem - especially in C/C++ that has limited introspection and massive pre-processing abuse to work around language limitations.

How rust crates interact with the Zephyr build system is going to be interesting to watch.