Add Apache Thrift Module (from GSoC 2022 Project)

[cfriedt]

Origin

Thrift is mostly ready for prime-time, with only a few more items on the Upstream checklist to complete. It would be great to get that into the 3.3 release.

Purpose

Zephyr participated in the Google Summer of Code 2022 under the umbrella of The Linux Foundation. This was already tabled and supported by the TSC in April, 2022.

The Thrift Zephyr module is a "lightweight wrapper" around the upstream Apache Thrift project.

The precise meaning of "lightweight wrapper" in this case, denotes that the Zephyr module only contains the minimal amount of CMake, YAML, Kconfig, and code to enable building the relevant portions of the Apache Thrift project so that it can be supported in Zephyr.

In fact, we have not copied or forked the upstream Apache Thrift project in any way and intend to track it over the long term: since the upstream Apache Thrift project is referenced as a git submodule, updating to more recent Thrift releases will be as simple as checking out a tagged release and making a PR to the Zephyr module.

Mode of integration

Merging Changes from gsoc-2022-thrift into zephyr

The gsoc-2022-thrift project consists mainly of glue code, samples, tests, and a cmake extension. Thrift itself is simply externalized. As such, integration with Zephyr is trivial.

• merge glue code directly into the zephyr tree
• reference the upstream Apache Thrift sources via west.yml

There are 2 options (FWICT) to merge the two projects:

1. Preserving all commit history is likely going to be a lot of effort, and would require formatting and applying each change via git patch-format and git am. There are 121 changes in the history of the gsoc-2022-thrift project.
2. It would be much easier to simply perform a git merge (with --allow-unrelated-histories). Copyright would be preserved, but commit history would not be preserved.

The cmake/, tests/, and samples/ directories can likely be merged as-is. However, zephyr/samples/ does not have a lib/ subdirectory, so samples/lib/thrift/ should be moved to samples/modules/thrift/.

The lib/thrift/ subdirectory incorporates Kconfig, CMakeLists.txt, as well as some glue code (.c, .cpp, .h files). Note, that the .h files cannot be named .hpp because they are overrides of upstream header files. Those workarounds will eventually be deleted once the Zephyr C++ subsystem supports std::thread, std::mutex, etc.

Incorporation of Zlib-Compatible API

While Thrift itself is licensed under the same terms as Zephyr (SPDX-License-Identifier: Apache-2.0), it requires the Zlib API for TZlibTransport, which transparently compresses and decompresses message contents. The de-facto zlib was not suitable for use in Zephyr due to size and symbol requirements.

Our GSoC contributor, Young Mei, was able to re-implement the zlib API by incorporating sources from uzlib (SPDX-License-Identifier: Zlib) and Inflater (SPDX-License-Identifier: MIT) into a convenient, 4-file implementation called "Muzic".

• defl_static.c
• genlz77.c
• inflater.c
• zlib_comp.c

These above sources are not considered as public API of the Thrift module and are more along the lines of implementation details. With that, if Zephyr were to ever consider adding a compression subsystem, with a Zlib-compatible API, then these sources could be removed in favour of using the added subsystem.

Integration of Apache Thrift Module

There are two options for incorporating the upstream Apache Thrift project into Zephyr via west.yml. It is not clear if there is specific policy that requires one or the other.

1. Create a fork of Apache Thrift under zephyrproject-rtos/

diff --git a/west.yml b/west.yml
index 166a76db88..449d8b397e 100644
--- a/west.yml
+++ b/west.yml
@@ -259,6 +261,9 @@ manifest:
     - name: zscilib
       path: modules/lib/zscilib
       revision: 0035be5e6a45e4ab89755b176d305d7a877fc79c
+    - name: thrift
+      path: modules/lib/thrift
+      revision: 9c0de2d1eb343910213c62325f73e3bb72361c22
 
   self:
     path: zephyr

• Pros:
  • we could do whatever we want with it
  • e.g. zephyr-ltsv3-branch
• Cons:
  • would require semi-regular sync with upstream
  • has the potential to diverge if not properly managed
  • yet another fork that sits in zephyrproject-rtos

4. Simply add apache/thrift as a remote

diff --git a/west.yml b/west.yml
index 166a76db88..449d8b397e 100644
--- a/west.yml
+++ b/west.yml
@@ -21,6 +21,8 @@ manifest:
   remotes:
     - name: upstream
       url-base: https://github.com/zephyrproject-rtos
+    - name: apache
+      url-base: https://github.com/apache
 
   #
   # Please add items below based on alphabetical order
@@ -259,6 +261,9 @@ manifest:
     - name: zscilib
       path: modules/lib/zscilib
       revision: 0035be5e6a45e4ab89755b176d305d7a877fc79c
+    - name: thrift
+      path: modules/lib/thrift
+      revision: 9c0de2d1eb343910213c62325f73e3bb72361c22
 
   self:
     path: zephyr

• Pros:
  • less duplication
  • less management overhead
  • could also host zephyr-specific branches like zephyr-ltsv3-branch
• Cons:
  • patches would need to be fixed upstream

Personally, I have been contributing to Apache Thrift for some time, and the process is relatively simple (see How to Contribute). My suggestion is to use the latter approach.

Which Version of Apache Thrift?

The specified revision above includes all required changes to apache/thrift already merged upstream. However, west.yml should be updated to point to the v0.18.0 release once that is tagged.

Maintainership

@cfriedt

Pull Request

#54013

For additional details, see Upstream Checklist

Description

Long description that will help reviewers discuss suitability of the
component to solve the problem at hand (there may be a better options
available.)

What is its primary functionality (e.g., SQLLite is a lightweight
database)?

Thrift is an IDL specification, RPC framework, and code generator. It works across all major operating systems, supports over 27 programming languages, 7 protocols, and 6 low-level transports. Thrift was originally developed at Facebook in 2007. Subsequently, it was donated to the Apache Software Foundation. Thrift supports a rich set of types and data structures, and abstracts away transport and protocol details, which lets developers focus on application logic.

What problem are you trying to solve? (e.g., a state store is
required to maintain ...)

The main problem that would be solved by using Thrift within Zephyr is lowering the barrier to enterprise-grade RPC for the average Zephyr developer.

Simply

1. define data types
2. define remote procedures
3. generate code
4. fill in the blanks.

Additionally, Thrift is flexible enough to be used across a variety of low-level transports, protocols, and programming languages meaning that it can adapt to virtually any Zephyr application requirement.

Why is this the right component to solve it (e.g., SQLite is small,
easy to use, and has a very liberal license.)

Thrift is the right choice because it is actively maintained by the ASF, under heavy use by organizations large and small across the globe, and was designed to be used on all operating systems.

Dependencies

At this time, the thrift module (https://github.com/zephyrproject-rtos/gsoc-2022-thrift) has no external dependencies outside of what is already available in Zephyr. Specifically:

• mbedTLS

Revision

Version or SHA you would like to integrate initially

License

Apache-2.0 for Thrift itself. Zlib-API (implementation detail) is MIT and Zlib licensed and can easily be swapped out to use a common compression / decompression API if such a subsystem is every made.

[cfriedt]

• removed the TSC label for now until the Upstream Checklist is complete