Consider a centralized repository for dependency management?

[fruffy]

Context: #3930 (comment)

Currently, our dependency initialization, most of which is done in CMake, is scattered across various top-level folders and the back ends. This makes it hard to supply your own dependencies and/or overwrite the hashes and URLs in the dependencies we pull. Further, it makes it difficult to comprehend where and when dependencies are actually being pulled in.

A centralized repository could be useful, but is tricky for two reasons. First, modularity. For example, the garbage collector dependency is only pulled in when ENABLE_GC is toggled. Protobuf and GTest are also meant to be optional. Second, every back end manages their own dependencies and we do not want to add back end dependencies to the central repository.

To implement some form of user-customizable dependencies we could introduce a JSON file listing all the necessary information on the dependencies. To parse this file there is string(JSON...). That file could also be user-provided. The problem with string(JSON...) is that it requires CMake 3.19+ and we are still working with CMake 3.16.

We could also just create a helper fetchcontent_declare_with_option which adds CMake options to set the URL, URL_HASH and GIT TAG for each fetchcontent_declare invocation.

[vlstill]

The problem with string(JSON...) is that it requires CMake3.19+ and we are still working with CMake3.16.

Note: We (= P4C) are currently supporting Ubuntu 20.04 which has 3.16 in the repositories. However, we are also building on 18.04 which is EOL already and we use user-istalled cmake there. I suggest we could drop 18.04, move the "main" path to be aligned to 22.04 and support 20.04 in the same way we do 18.04 now. 20.04 will EOL within a year, so it would make sense to me to move to something newer (while still keeping it supported of course).

So personally I have no problem with moving to CMake 3.19. At Altera we currently use 3.20 and will likely update to even newer CMake so we will not be impacted. This might be worth it regardless of the dependency management.

We could also just create a helper fetchcontent_declare_with_option which adds CMake options to set the URL, URL_HASH and GIT TAG for each fetchcontent_declare invocation.

I think that would work too, if there was an option to provide the built package locally too (even in a form of a CMake module that is responsible for setting all the cmake targets and variables -- I don't really see how this could be done in a simpler way).

Even simpler, the FetchContent is already designed with overrides from higher-level projects in mind, see https://cmake.org/cmake/help/latest/module/FetchContent.html#complex-dependency-hierarchies. I don't know if there is a way of using this to also refer to a locally-available dependency instead of the fetched one. If there is, maybe this would be sufficient. The downside is that this would still mean the information is scattered around the CMakeLists.txt files, but the advantage would be it would retain the ability to pull just the needed dependencies based on the configuration of P4C. If we go this way, we should document how to override dependencies in the downstream pojects.

If we wanted to centralize it, we could maybe have one cmake file with functions for adding each of the dependencies, then call these function when the target is actually used.

[fruffy]

Do you know a user-friendly, local way to install CMake with a more recent version? That is my main concern.

It might be worthwhile to consider Conan at this point. It could simplify our dependency management a lot. I am currently investigating modular boost and it is a massive pain even with FetchContent.

Conan or vcpkg could give us central dependency management. Although I am not sure if they can be modularized for the various back ends.

[jafingerhut]

"Do you know a user-friendly, local way to install CMake with a more recent version? That is my main concern."

Do you mean, install a more recent version of CMake itself? Of course it can be built from whatever version of its source code that you like. I don't know what storage resources we have available, but building a particular version of CMake for various processor/OS combinations, and storing them somewhere for the p4lang project, seems potentially useful to avoid building it from source repeatedly. The same approach might apply for other programs and libraries used in building or testing p4c.

[fruffy]

Do you mean, install a more recent version of CMake itself? Of course it can be built from whatever version of its source code that you like. I don't know what storage resources we have available, but building a particular version of CMake for various processor/OS combinations, and storing them somewhere for the p4lang project, seems potentially useful to avoid building it from source repeatedly. The same approach might apply for other programs and libraries used in building or testing p4c.

My thinking was something like Python virtual environment. Ideally, we want to build our compiler using a localized version of CMake which does not interfere with whatever system-level version a user may have. That would make it easier for us to change the minimum CMake version as necessary.

[vlstill]

Do you know a user-friendly, local way to install CMake with a more recent version? That is my main concern.

python3 -m pip install --user cmake==3.29.3

seems to work just fine. This is the way we already recommend for old Ubuntu. Since this is already using pip, you can also use it to install to venv (just drop the --user):

$ python -m venv cmake
$ . ./cmake/bin/activate
$ python3 -m pip install  cmake==3.29.3
$ $ ./cmake/bin/cmake --version
cmake version 3.29.3

So yes, this is possible and quite easy. The python bit is kind of superfluous (cmake is just a python wrapper that presumably executes $VENV_DIR/lib/python3.9/site-packages/cmake/data/bin/cmake in the end. But that does not matter much in practice.

[vlstill]

Do you know a user-friendly, local way to install CMake with a more recent version? That is my main concern.

It might be worthwhile to consider Conan at this point. It could simplify our dependency management a lot. I am currently investigating modular boost and it is a massive pain even with FetchContent.

Conan or vcpkg could give us central dependency management. Although I am not sure if they can be modularized for the various back ends.

As a downstream project developer, I would be frankly much more comfortable with cmake-only solution. I am not very familiar with neither Conan nor vcpkg, but I would expect it to bring extra complications, especially if use of an alternative version (to the P4C-defined) of a dependency is required.

[hanw]

Take a look at this project? https://github.com/cpm-cmake/CPM.cmake

[fruffy]

Take a look at this project? https://github.com/cpm-cmake/CPM.cmake

This looks interesting. It has a couple convenience functions over FetchContent and supports package locks. CPM_USE_LOCAL_PACKAGES also implements what we are currently doing manually.