[processor/tailsampling] Late span age histogram should include sampled traces

[portertech]

Currently, the late span age histogram only includes non-sampled traces. The histogram should also include sampled traces. This pull request includes anything with a decision time.

Note: The decision cache still wrecks it 😅

[fatsheep9146]

Is it possible to add a unit test for this fix?

[portertech]

Is it possible to add a unit test for this fix?

I'll give it a go 👍

[portertech]

@fatsheep9146 I adjusted the existing histogram test to touch sampled traces 👍

I confirmed the test fails on main, only half of the late spans (not sampled) are counted:

?       github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor/internal/telemetry    [no test files]
--- FAIL: TestProcessorTailSamplingSamplingLateSpanAge (0.00s)
    processor_telemetry_test.go:519: [Metrics Data not equal: Histogram not equal: Histogram DataPoints not equal:
        missing expected values:
        metricdata.HistogramDataPoint[int64]{Attributes:attribute.Set{equivalent:attribute.Distinct{iface:interface {}(nil)}}, StartTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Time:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Count:0xa, Bounds:[]float64{0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000}, BucketCounts:[]uint64{0xa, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Min:metricdata.Extrema[int64]{value:0, valid:true}, Max:metricdata.Extrema[int64]{value:0, valid:true}, Sum:0, Exemplars:[]metricdata.Exemplar[int64](nil)}
        unexpected additional values:
        metricdata.HistogramDataPoint[int64]{Attributes:attribute.Set{equivalent:attribute.Distinct{iface:[0]attribute.KeyValue{}}}, StartTime:time.Date(2025, time.January, 13, 22, 55, 4, 84030000, time.Local), Time:time.Date(2025, time.January, 13, 22, 55, 4, 84311000, time.Local), Count:0x5, Bounds:[]float64{0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000}, BucketCounts:[]uint64{0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Min:metricdata.Extrema[int64]{value:0, valid:true}, Max:metricdata.Extrema[int64]{value:0, valid:true}, Sum:0, Exemplars:[]metricdata.Exemplar[int64]{}}
        ]

[portertech]

Looks like an unrelated lint error.

[jpkrohling]

This LGTM, but I really, really think the tail-sampling processor is due to a refactor. "Late span" used to refer to spans that arrived after a sampling decision had been made and dropped, so that people could see why traces are incomplete. With the decision caches, the whole idea of late spans need to be reconsidered. Is it still valuable to track it? Why? If so, it should be better documented.

[portertech]

This LGTM, but I really, really think the tail-sampling processor is due to a refactor. "Late span" used to refer to spans that arrived after a sampling decision had been made and dropped, so that people could see why traces are incomplete. With the decision caches, the whole idea of late spans need to be reconsidered. Is it still valuable to track it? Why? If so, it should be better documented.

I agree, it is due for a refactor. This PR, along with #37212 and #37189 are immediate improvements in preparation for such a refactor. Next, we can require the decision caches (breaking change) and optimize the current architecture (e.g. batcher and decision tick loop). It's a very small leap which will result in considerable resource savings. From there, we can gather enough data to direct more significant rework.