Add config for redis data expiration

[jefimm]

Currently writing data to redis does not set any TTL on the data - this will eventually fill the disk
Please add configuration for the expiration of the data written to redis with default of no expiration

[Issif]

Good point. I will add it. Thanks

[Issif]

I dug to add this feature, and for now, it's not possible. The module I use for communication with redisearch is https://github.com/RediSearch/redisearch-go (official SDK) and it doesn't allow to send an FT.EXPIRE command to set a ttl.
I think it's not a big deal to submit a PR for adding this feature in the SDK but I don't know how long it will take before the maintainers release a new version.

[Issif]

I added 500k (507457) events in a local redisearch container, here's the details about memory usage:

# Memory
used_memory:1347273984
used_memory_human:1.25G
used_memory_rss:1366188032
used_memory_rss_human:1.27G
used_memory_peak:1347275552
used_memory_peak_human:1.25G
used_memory_peak_perc:100.00%
used_memory_overhead:25562504
used_memory_startup:904848
used_memory_dataset:1321711480
used_memory_dataset_perc:98.17%
allocator_allocated:1347414104
allocator_active:1349976064
allocator_resident:1373999104
total_system_memory:33355489280
total_system_memory_human:31.06G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:2561960
allocator_rss_ratio:1.02
allocator_rss_bytes:24023040
rss_overhead_ratio:0.99
rss_overhead_bytes:-7811072
mem_fragmentation_ratio:1.01
mem_fragmentation_bytes:18955064
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:165040
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

I don't know your use-case, but if you have 500k events a year, maybe your rules are not optimum and create flood.

[jefimm]

I have installed falco + falcosidekick-ui on our test cluster before tuning and adjusting rules and it run out of 1GB disk space in less than a week. This is the reason for this request.

[Issif]

How many events did you get for 1GB? I get your point, it's a really valuable idea and I would like to implement the feature, but the SDK doesn't allow it, here's the function they use to add keys https://github.com/RediSearch/redisearch-go/blob/bd32e0b428fd340ac1bc71812e9e9dcdcc852e52/redisearch/query.go#L417/query.go#L417

[jefimm]

maybe this would help https://stackoverflow.com/questions/66028975/how-to-set-a-key-with-expiry-using-redisgo-api

[Issif]

redisearch-go sdk uses redisgo as background, I could use it also but it means to deal with 2 different clients, which is not a good pattern. the best should be to submit a PR to redisearch to add the feature

[GauJosh]

@Issif How to use this setting when setting up sidekick-ui with helm chart? Is there a "ttl" field which can be set inside values.yaml?

[Issif]

The helm chart is not ready as this feature will be in next falcosidekick-ui release. But yes, an helm value will expose it.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale