Lib-common

The lib-common is an Intersec libc extension targeting Linux environments. It provides most of the basic tools a C programmer can expect such as:

• a string manipulation library

• containers like dynamic vectors, hash tables, hash maps, lists…​

• bitmask manipulation functions

• endianness conversion primitives

• …​

It also contains higher level libraries:

• asynchronous event loop

• http client and server implementations

• RPC communication channels

• optimized allocators for our use cases

• log file management

• tracing and reporting primitives

• program argument parser

• ASN1 parser / writer

• A prometheus client

• …​

It is also the home of our IOP library. The IOP (Intersec Object Packer) is a framework to serialize structured data to use in communications protocols, data storage…​ IOP is language independent. IOP objects are encoded using TLV (Tag-Length-Value) packers/un-packers. The IOP concept is similar to the Google Protocol Buffers.

The IOP library is used to transmit data over the network in a safe manner. It deals with data integrity checking, retro-compatibility issues,…​ They are also used to exchange data between different languages or to provide a generic interface to store and load C data on disk.

Documentation

The lib-common documentation can be found in the docs directory. The documentation is based on Antora, an online version is available here.

Quickstart

This section offers basic instructions for building the lib-common.

Prerequisites

The lib-common is mainly written in C, with some extensions coming from clang features. So its main dependencies are:

glibc

used extensively, in particular to interface with various Linux kernel facilities

llvm

base compiler toolchain

libclang-dev

base c and cpp compiler development library

clang

base c and cpp compiler

waf

build system

poetry

managing Python dependencies

openssl

various cryptography requirements

flex

IOP syntax lexer at https://github.com/westes/flex

gperf

GNU library for perfect hash function generation at https://www.gnu.org/software/gperf

Most of dependencies can be installed using your own Linux distribution packages, with the exception of waf which is usually not packaged.

waf installation

The lib-common build system uses waf. If you want to understand how it works, you should read the Waf Book, but this is not necessary if you just want to use it.

All the lib-common specificities and custom rules are defined in waf tools, located in the build/waftools directory.

The main entry point of the build system is the wscript file located at top-level of the repository. It loads the Intersec tools, defines the main configure and build functions, and recurses in sub-directories (that contain their own wscript or wscript_build files).

The waftools and wscript files are supposed to be readable and documented when needed.

waf can be downloaded as a single executable or built from sources, as described in the waf documentation.

The simplest approach is to download the waf executable and install it in a directory already in your PATH environment, for example /usr/local/bin:

$ cd /usr/local/bin
$ wget https://waf.io/waf-2.0.x
$ ln -s waf-2.0.x waf
$ chmod +x waf-2.0.x
$ ./waf

waf versions 2.0.8 and 2.0.18 are known to work fine, running the latest 2.0.x waf should also work fine.

Dependencies

List of packages per Linux distribution

Debian 9+ or Ubuntu 18.04+

You need to install the following packages to build the lib-common:

$ sudo apt-get install clang libclang-dev llvm-dev python3-dev python3-pip flex gperf pkg-config exuberant-ctags libxml2-dev libssl-dev

Optionally, you can also install the following packages for a complete installation:

$ sudo apt-get install exuberant-ctags valgrind

In order to run the tests, the following packages also need to be installed:

$ sudo apt-get install smitools snmp-mibs-downloader

Fedora 31+

You need to install the following packages to build the lib-common:

$ sudo dnf install clang clang-devel llvm-devel flex gperf libxml2-devel openssl-devel python3-devel

Optionally, you can also install the following packages for a complete installation:

$ sudo dnf install ctags-etags valgrind-devel

In order to run the tests, the following packages also need to be installed:

$ sudo dnf install libsmi diffutils

Poetry

Poetry is used to install Python dependencies used on compilation (Cython), and in tests. To install Poetry, follow the instructions on the official documentation: https://python-poetry.org/docs/#osx—​linux—​bashonwindows-install-instructions

Building lib-common

First of all, you have to configure your project, by running in the top-level directory:

$ waf configure

By default, it will use gcc. You can also use clang by defining the two environment variables CC and CXX:

$ CC=clang CXX=clang++ waf configure

If the configuration step triggers no error, then you are ready to build, which just consists in running:

$ waf
or
$ waf build

You can run it from a sub-directory in order to build only the targets defined in this directory and its sub-directories (and its dependencies).

All the available targets can be listed with this command:

$ waf list

It is possible to build only a specific target, or a list of targets, by running, from anywhere in the repository:

$ waf --targets=target1,target2

Other Intersec-specific waf commands

Other commands are listed with waf --help. Here they are:

• waf check: run the tests of the current directory (defined in the ZFile) and in its sub-directories. The following variants also exist (cf waf --help for the details): fast-check, www-check, selenium, fast-selenium.

• waf tags: generate tags using ctags.

• waf etags: generate tags for emacs using ctags.

• waf pylint: run pylint checks on committed python files.

• old-gen-files-detect: detect old files generated by a previous build system run.

• old-gen-files-delete: delete the files detected by the previous command.

• coverage-start: start a coverage session (requires coverage profile). This resets the coverage counters. After running this command, you can run some code and use the coverage-end command to produce a coverage report. Note that this is done when configuring the project.

• coverage-end: end a coverage session and produce a report.

Supported environment variables

The following environment variables can be used at the configuration phase:

P (string)

Specify the desired compilation profile (default, debug, release, …​). The complete list of available profiles is defined in build/waftools/backend.py, variable PROFILES. If not specified, the default profile is default.

NOCHECK (boolean)

The build-system doesn’t run "check" targets, which are:

• clang check of c files.

• linters on js/ts files. You may want to set it to speedup the build. You can also bypass the checks thanks to the nocheck parameter of task-generators, which can be True to bypass the checks of all the source files, or a list of files to not check.

NOASSERT (boolean)

If set, the assertions and debug-related code won’t be compiled. By default, assertions are disabled in release and mem-bench profiles, and enabled in all the others.

NO_DOUBLE_FPIC (boolean)

If set, the compilation will be faster, but the produced binaries will be larger and the runtime performances will be affected. Cf. build/waftools/backend.py for the details. This is ignored in release profile.

FAKE_VERSIONS (boolean)

If set, the version files are generated with fake (and constant) data, so that changing of git revision does not trigger a re-link of all the binaries. This is a huge gain of time, but it’s not possible to know the revision of the binaries that are built with this flag. This is ignored in release profile.

SHARED_LIBRARY_SANITIZER (boolean)

If set, the shared libraries will also be compiled with the sanitizer specified by the profile. You will have to use LD_PRELOAD or use a process that is compiled with the same sanitizer to load the compiled shared libraries with this option. This is only available for profiles that use sanitizers, i.e. asan or tsan.

Compiling with gcc

An important part of the lib-common uses the blocks clang extension, that is not supported by gcc.

Because of this, we have put in place a two-phase build of some files (named foo.blk instead of foo.c) that are pre-compiled using a special tool named clang-rewrite-blocks. gcc is then used to produce the final object code.

clang-rewrite-blocks is compiled and used automatically by the build-system. See src/`clang-rewrite-blocks/README.adoc` for more information about clang-rewrite-blocks.

Contributing

In the spirit of open source software, everyone is welcome to contribute to this project!

The best way to get involved is to just show up and make yourself heard. We pride ourselves on having a very friendly and encouraging culture. Whether you’re a user, writer, designer, developer, architect, devops, system administrator, advocate, project manager, or just someone with an idea about how to improve the project, we welcome your participation. In return, you’ll get to use better software that we built together as a community.

Thanks in advance for helping to make this project a success!

Copyright and License

Copyright © 2005-2022 by Intersec SA and the individual contributors to lib-common.

Licensed under the Apache License, Version 2.0 (the "License"). You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Authors

Development of the lib-common is led and sponsored by Intersec.