Upstreaming Snabbwall app #1116
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This implements a small scripts which parses the "ndpi_protocol_ids.h" header (which is part of the public API of nDPI) and generates a Lua module with the parsed protocol, service, and content identifiers used by the nDPI library. Also included there is a version of "ndpi/protocol_ids.lua" generated using the script, using the header for version 1.6 of nDPI.
This implements a small scripts which parses the "ndpi_protocol_ids.h" header (which is part of the public API of nDPI) and generates a Lua module with the parsed protocol, service, and content identifiers used by the nDPI library. Also included there is a version of "ndpi/protocol_ids.lua" generated using the script, using the header for version 1.6 of nDPI.
Generate ndpi/protocol_ids.lua from nDPI headers
Initial pass at a binding for nDPI using the LuaJIT FFI, containing the
following modules:
ndpi.c
Low-level FFI definitions of the underlying C data structures and
functions. In general, not to be used by client code directly.
ndpi.protocol_bitmask
Contains a pure-Lua definition of the NDPI_PROTOCOL_BITMASK type.
nDPI defines this type and operations on it using the C preprocessor,
which the LuaJIT FFI does not implement. This is implemented in a way
that it is possible to easily adapt the code if the nDPI definition
changes, by tweaking the constants at the beginning of the file.
ndpi.wrap
Defines convenient wrappers for the low-level FFI definitions using
"ffi.metatype" and manually wrapping some of the functions to provide
a more idiomatic API. This also loads ndpi.protocol_bitmask and
ndpi.protocol_ids and exposes them as part of the API.
ndpi
The top-level module simply loads the wrapper module.
At the moment the binding requires nDPI version 1.7 and has not been tested
with any other versions.
The "examples/readpcap" script implements an example on how to use the "ndpi" module to implement a tool similar to the "pcapReader" one included with nDPI in Lua. The main difference is that only reading packets from a pcap capture is implemented (live capture is not supported), and the statistics are printed differently. As a freebie, the "examples.pcap" module is provided, which implements a very minimal wrapper over libpcap which can be used to iterate over the packets of a pcap file.
When the READPCAP_DEBUG environment variable is set to a value different than zero, information about packets is printed as each packet is processed.
This solves the issue which made examples/readpcap always show source and destination ports as zero.
Instead of always using the supplied example pcap capture, allow specifying one as the first argument to the script.
This also displays the number of flows detected and identified.
Provided that ljndpi works with LuaJIT, this change hints the user that stock Lua is not supported. Coincidentaly, note that tools/update-protocol-ids does work with stock Lua, so "lua" is kept in its shebang.
This prevented using the "strict.lua" module, which is always enabled in some applications (e.g. Snabb Switch). In such cases, just loading the module would cause the application to raise an error when trying to access the undefined variable.
This also includes skeleton for the "snabb wall pcap" subcommand.
The variables "id_free" and "flow_free" were undefined, so the calls to "ffi.gc()" were being passed effectively a "nil" as second argument.
For unknown reasons, the `newp == false` check triggers a segfault when running Snabbwall in certain cases. This change to the conditional guard is a workaround until the deeper issue is fixed.
Improves performance of `snabb wall spy` by a good amount
This improves performance quite a bit by eliminating an allocation that causes a trace abort. The caveat is that Snabbwall must ensure that the lifetime of the flow keys are short, so that conflicting writes do not happen.
Dynasm is used to construct a wrapper function that avoids a struct return (that LuaJIT does not support for traced FFI calls) by using a struct pointer. We use dynasm to avoid having to compile a C wrapper which complicates the build process. The wrapper isn't expressible in Lua.
Optimizations for snabbwall & benchmarking script
Closes Snabbwall issue #16
Undoes some Makefile changes that were specific to Snabbwall development that don't make sense for upstream
Usually Snabbwall has tracked release versions of Snabb. I'm merging in the `next` branch instead now in order to make it easier to upstream Snabbwall without merge conflicts in the pull request. Conflicts: lib/pflua/src/pf/anf.lua lib/pflua/src/pf/match.lua
|
Note: I updated snabbwall by merging in |
|
Thank you. @aperezdc has agreed to change the ljndpi license to Apache 2.0 from MIT as well. If anyone in the Snabb community has any reservations about Snabb containing a wrapper for an LGPLv3 library (ndpi), please speak up quickly. :-) |
One of the main motivations of ljndpi is to be used as part of SnabbWall, a suite of Layer 7 firewall components for Snabb, which itself is Apache-licensed. Using the same license as Snabb makes it simpler to incorporate as part of it.
|
I have just pushed version 0.1.0 of ljndpi. Despite the version bump, the functionality is the same as the previous ( Edit: Given that |
|
@aperezdc thanks, the branch should be updated with your license change now. |
|
@takikawa the selftest fails on SnabbBot, presumably due to missing dependencies. Is there a list of those somewhere so I can add them to snabb_bot.nix? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a PR for upstreaming Snabbwall for review by @kbara for
kbara-next(the PR targetsnextfor CI purposes). The downstream branch now lives atIgalia/snabb(now a protected branch too) and I've added a commit that updatesdoc/branches.mdas well.