feat: Update in-memory trace cache to use LRU instead of ring buffer

[MikeGoldsmith]

Which problem is this PR solving?

Updates the InMemCache cache implementation to use uses a hasicorp/golang-lru cache to order traces. The cache checks if the capacity is set to the default cache size (10000), and replaces the capacity with math.MaxInt32 instead.

This means the cache will only contain active traces and consume only the required bytes as expired because old traces are removed instead of lingering in the buffer. The collector regularly checks the Refinery process's memory usage, and will shed old traces if over the configured allocation then manually called GC to clean up used resources.

Because Refinery applies a consistent trace timeout, the LRU enables us to efficiently retrieve the oldest trace when evicting expired traces unlike the ring buffer than required us to loop over the entire buffer when searching for expired traces or removing sent traces.

Below is a comparison of the cache before and after the update

• Closes Revisiting trace cache strategy #1339

Short description of the changes

• Add new trace cache, backed by an LRU cache
• Update benchmarks to compare current and new cache implementations for various actions (Set, Get, Remove, etc)

---

Before

BenchmarkCache_Set/InMemCache-10           	   	1000000000	         0.0000008 ns/op	       0 B/op	       0 allocs/op
BenchmarkCache_Get/InMemCache-10          	    	1000000000	         0.0000004 ns/op	       0 B/op	       0 allocs/op
BenchmarkCache_RemoveTraces/InMemCache-10              	1000000000	         0.0000335 ns/op	       0 B/op	       0 allocs/op
BenchmarkCache_TakeExpiredTraces/InMemCache-10         	   18817	     63078 ns/op	      50 B/op	       1 allocs/op

After

BenchmarkCache_Set/TraceCache-10           	   	1000000000	         0.0000004 ns/op	       0 B/op	       0 allocs/op
BenchmarkCache_Get/TraceCache-10          	    	1000000000	         0.0000003 ns/op	       0 B/op	       0 allocs/op
BenchmarkCache_RemoveTraces/TraceCache-10              	1000000000	         0.0000015 ns/op	       0 B/op	       0 allocs/op
BenchmarkCache_TakeExpiredTraces/TraceCache-10         	27758473	        45.23 ns/op	      41 B/op	       0 allocs/op

Note the updated cache is considerable faster removing traces (x10 faster) and removing expired traces (> x1000).

[kentquirk]

I haven't even looked at the implementation, but why do we even want an LRU any more? There's no reason to limit the number of traces in the cache; the size of the trace cache is basically a rounding error in the total memory usage.

You weren't in the conversation, but when I talked to Yingrong about this, we talked about simply letting the cache size adjust as necessary. It's the total memory usage that actually matters.

[VinozzZ]

Why do we change this to 1 second?

[MikeGoldsmith]

The non-sampled event is added and we only waits 5 seconds for it to appear in the transmission queue. The cache will only expire and send the events once the trace timeout is reached.

I'm not sure how / why it currently works in main.

[VinozzZ]

Oh, it's because the cache size is set to 3 in the cache.NewInMemCache(). Once the cache is full, traces will be ejected from the cache when a new span comes in and put into the transmission queue.