refactor(userspace/engine): reduce memory usage for resolving evttypes #1965
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jasondellaluce
force-pushed
the
refactor/evttypes
branch
from
8feb152 to
1740394
Compare
mstemm
reviewed
Apr 6, 2022
jasondellaluce
force-pushed
the
refactor/evttypes
branch
from
1740394 to
978f684
Compare
|
I reverted the refactorings of the |
|
/milestone 0.32.0 |
jasondellaluce
force-pushed
the
refactor/evttypes
branch
2 times, most recently
from
4478ebb to
a727440
Compare
mstemm
requested changes
Apr 14, 2022
…ilters and event names Signed-off-by: Jason Dellaluce <[email protected]>
…nsinsp Signed-off-by: Jason Dellaluce <[email protected]>
At the same time, this also simplifies the unit test cases by using the SCENARIO construct of catch2, which allows sharing a setup phases between different unit tests, and removes a bunch of repeated LOC in our case. Signed-off-by: Jason Dellaluce <[email protected]>
…e loader Signed-off-by: Jason Dellaluce <[email protected]>
jasondellaluce
force-pushed
the
refactor/evttypes
branch
from
fd2f02e to
032a959
Compare
Signed-off-by: Jason Dellaluce <[email protected]>
Signed-off-by: Jason Dellaluce <[email protected]>
Signed-off-by: Jason Dellaluce <[email protected]>
jasondellaluce
force-pushed
the
refactor/evttypes
branch
from
032a959 to
f613c26
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jasondellaluce, leogr The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
mstemm
requested changes
Apr 19, 2022
mstemm
approved these changes
Apr 19, 2022
|
LGTM label has been added. Git tree hash: e081a992c33c2d418293c2bd72a3cb0ee8bf9958
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
/kind cleanup
Any specific area of the project related to this PR?
/area engine
/area tests
What this PR does / why we need it:
In falcosecurity/libs#74 we added a feature in libsinsp for searching the event types for a given filtercheck. The set of event types is used internally by the Falco rule engine to index each rule filter by event type to improve performance at runtime. Before this, we used to calculate the set of event types right from Falco.
In the implementation of falcosecurity/libs#74, filterchecks maintain the set of evttypes as part of the internal class state. Since this information is used only at initialization time, maintaining it at runtime causes a memory abuse that ended up in Falco 0.31+ using way more memory than needed, with the limit increasing depending on how many rules are loaded in the engine.
This PR fixes this by re-introducing the evttype search back into Falco, this time leveraging the new AST definitions introduced in falcosecurity/libs#217. We do so by implementing a simple AST visitor class that populates the set of evttypes for a given filter condition. I think this is optimal because this use case is only related to Falco and I don't think the logic should be part of libsinsp. Moreover, this allows a better control (for example, we want to find evttypes only for the
syscallsource whereas this is handled differently fork8s_auditand plugins).This also deprecates the changes of falcosecurity/libs#74, that can be removed after this PR gets merged (see falcosecurity/libs#280). The unit tests related to falcosecurity/libs#74 have been ported from libsinp to Falco and adapted to the Catch2 framework.
Here's some data about Falco's memory usage. The following was measured at runtime by after launching Falco by simply loading the
falco_rules.yamldefault ruleset:Which issue(s) this PR fixes:
Not sure this is a direct fix, but this can help for the following issues that reported Falco consuming more memory at runtime:
Special notes for your reviewer:
Of the 650+ LOC added in this PR, but the real contribution is userspace/engine/filter_evttype_resolver.cpp with only ~160 LOC, the rest are just renamings and unit tests ported from libsinsp.
Does this PR introduce a user-facing change?: