[elasticsearchexporter] Direct serialization without objmodel in OTel mode

[felixbarny]

Directly serializes pdata to JSON in OTel mode

• Improved performance as no objmodel.Document needs to be created first
• Fixes issue discovered in [elasticsearchexporter] Support for complex attributes for log records in OTel mode #37021 where map bodies with dotted field names are de-dotted

[felixbarny]

Benchmark results:

TL;DR: the throughput is over 2x for metrics and over 3x for logs and traces. The allocated bytes/op are reduced 82% for metrics and 95% for logs and traces.

 goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter/integrationtest
                                      │   old.txt   │            new_final.txt            │
                                      │   sec/op    │   sec/op     vs base                │
Exporter/logs/otel/small_batch-10       79.16µ ± 1%   26.69µ ± 1%  -66.29% (p=0.000 n=10)
Exporter/logs/otel/medium_batch-10      757.0µ ± 2%   225.9µ ± 3%  -70.15% (p=0.000 n=10)
Exporter/logs/otel/large_batch-10       7.392m ± 1%   2.075m ± 0%  -71.93% (p=0.000 n=10)
Exporter/logs/otel/xlarge_batch-10      70.50m ± 1%   20.33m ± 1%  -71.17% (p=0.000 n=10)
Exporter/metrics/otel/small_batch-10    414.8µ ± 1%   181.0µ ± 0%  -56.37% (p=0.000 n=10)
Exporter/metrics/otel/medium_batch-10   3.960m ± 0%   1.717m ± 1%  -56.63% (p=0.000 n=10)
Exporter/metrics/otel/large_batch-10    39.97m ± 1%   18.12m ± 0%  -54.67% (p=0.000 n=10)
Exporter/metrics/otel/xlarge_batch-10   421.3m ± 1%   187.0m ± 1%  -55.61% (p=0.000 n=10)
Exporter/traces/otel/small_batch-10     79.64µ ± 0%   26.66µ ± 1%  -66.52% (p=0.000 n=10)
Exporter/traces/otel/medium_batch-10    765.5µ ± 1%   227.4µ ± 0%  -70.29% (p=0.000 n=10)
Exporter/traces/otel/large_batch-10     7.341m ± 1%   2.102m ± 0%  -71.37% (p=0.000 n=10)
Exporter/traces/otel/xlarge_batch-10    71.74m ± 1%   20.76m ± 0%  -71.06% (p=0.000 n=10)
geomean                                 4.171m        1.426m       -65.80%

                                      │   old.txt   │            new_final.txt             │
                                      │  events/s   │  events/s    vs base                 │
Exporter/logs/otel/small_batch-10       126.3k ± 1%   374.7k ± 1%  +196.62% (p=0.000 n=10)
Exporter/logs/otel/medium_batch-10      132.1k ± 2%   442.6k ± 3%  +235.04% (p=0.000 n=10)
Exporter/logs/otel/large_batch-10       135.3k ± 1%   481.9k ± 0%  +256.21% (p=0.000 n=10)
Exporter/logs/otel/xlarge_batch-10      141.8k ± 1%   492.0k ± 1%  +246.84% (p=0.000 n=10)
Exporter/metrics/otel/small_batch-10    168.8k ± 1%   386.8k ± 0%  +129.18% (p=0.000 n=10)
Exporter/metrics/otel/medium_batch-10   176.7k ± 0%   407.6k ± 1%  +130.60% (p=0.000 n=10)
Exporter/metrics/otel/large_batch-10    175.1k ± 1%   386.4k ± 0%  +120.62% (p=0.000 n=10)
Exporter/metrics/otel/xlarge_batch-10   166.1k ± 1%   374.3k ± 1%  +125.29% (p=0.000 n=10)
Exporter/traces/otel/small_batch-10     125.6k ± 0%   375.0k ± 1%  +198.67% (p=0.000 n=10)
Exporter/traces/otel/medium_batch-10    130.6k ± 1%   439.7k ± 0%  +236.56% (p=0.000 n=10)
Exporter/traces/otel/large_batch-10     136.2k ± 0%   475.8k ± 0%  +249.27% (p=0.000 n=10)
Exporter/traces/otel/xlarge_batch-10    139.4k ± 1%   481.6k ± 0%  +245.48% (p=0.000 n=10)
geomean                                 145.0k        424.1k       +192.44%

                                      │    old.txt    │            new_final.txt             │
                                      │     B/op      │     B/op      vs base                │
Exporter/logs/otel/small_batch-10       80.579Ki ± 0%   4.227Ki ± 0%  -94.75% (p=0.000 n=10)
Exporter/logs/otel/medium_batch-10      793.10Ki ± 0%   32.21Ki ± 0%  -95.94% (p=0.000 n=10)
Exporter/logs/otel/large_batch-10       7912.7Ki ± 0%   306.5Ki ± 0%  -96.13% (p=0.000 n=10)
Exporter/logs/otel/xlarge_batch-10      77.155Mi ± 0%   2.908Mi ± 0%  -96.23% (p=0.000 n=10)
Exporter/metrics/otel/small_batch-10    403.89Ki ± 0%   68.43Ki ± 0%  -83.06% (p=0.000 n=10)
Exporter/metrics/otel/medium_batch-10   4020.0Ki ± 0%   660.2Ki ± 0%  -83.58% (p=0.000 n=10)
Exporter/metrics/otel/large_batch-10    39.512Mi ± 0%   6.842Mi ± 0%  -82.68% (p=0.000 n=10)
Exporter/metrics/otel/xlarge_batch-10   390.35Mi ± 0%   65.27Mi ± 0%  -83.28% (p=0.000 n=10)
Exporter/traces/otel/small_batch-10     80.745Ki ± 0%   5.163Ki ± 0%  -93.61% (p=0.000 n=10)
Exporter/traces/otel/medium_batch-10    794.59Ki ± 0%   40.78Ki ± 0%  -94.87% (p=0.000 n=10)
Exporter/traces/otel/large_batch-10     7924.7Ki ± 0%   391.5Ki ± 0%  -95.06% (p=0.000 n=10)
Exporter/traces/otel/xlarge_batch-10    77.343Mi ± 0%   3.785Mi ± 0%  -95.11% (p=0.000 n=10)
geomean                                  4.219Mi        311.7Ki       -92.79%

                                      │   old.txt   │            new_final.txt            │
                                      │  allocs/op  │  allocs/op   vs base                │
Exporter/logs/otel/small_batch-10        553.0 ± 0%    112.0 ± 0%  -79.75% (p=0.000 n=10)
Exporter/logs/otel/medium_batch-10      5.430k ± 0%   1.018k ± 0%  -81.25% (p=0.000 n=10)
Exporter/logs/otel/large_batch-10       54.19k ± 0%   10.07k ± 0%  -81.41% (p=0.000 n=10)
Exporter/logs/otel/xlarge_batch-10      541.7k ± 0%   100.5k ± 0%  -81.44% (p=0.000 n=10)
Exporter/metrics/otel/small_batch-10    4.301k ± 0%   1.567k ± 0%  -63.57% (p=0.000 n=10)
Exporter/metrics/otel/medium_batch-10   42.83k ± 0%   15.49k ± 0%  -63.83% (p=0.000 n=10)
Exporter/metrics/otel/large_batch-10    428.0k ± 0%   154.8k ± 0%  -63.83% (p=0.000 n=10)
Exporter/metrics/otel/xlarge_batch-10   4.278M ± 0%   1.546M ± 0%  -63.86% (p=0.000 n=10)
Exporter/traces/otel/small_batch-10      594.0 ± 0%    132.0 ± 0%  -77.78% (p=0.000 n=10)
Exporter/traces/otel/medium_batch-10    5.830k ± 0%   1.218k ± 0%  -79.11% (p=0.000 n=10)
Exporter/traces/otel/large_batch-10     58.19k ± 0%   12.07k ± 0%  -79.26% (p=0.000 n=10)
Exporter/traces/otel/xlarge_batch-10    581.7k ± 0%   120.5k ± 0%  -79.28% (p=0.000 n=10)
geomean                                 35.09k        8.569k       -75.58%

[axw]

LGTM. The amount of handwritten serialisation makes me a little uncomfortable, but we can perhaps improve that with code generation later.

[felixbarny]

I've introduced pooling for the buffer holding the serialized events in 5e523c5. This reduced allocation by another 60% and increased throughput as well. I suppose most of the remaining allocations are from creating the pdata model itself, something that we can't easily optimize and not allocations that are directly caused by the ES exporter itself. I've updated the benchmark results in #37032 (comment).

[felixbarny]

4277f76 and 2328a7a add some more minor optimizations that have an average change to the previous commit of +9% events/s and -6.75% B/op.

I have another change lined up to mark the component with consumer.Capabilities{MutatesData: false}, which avoids the collector having to clone all input data before calling the exporter. But I'd like to get this PR in first, as it's already quite large.

[carsonip]

lgtm, thanks! Perf gain is wonderful.

[ChrsMark]

@JaredTan95 @lahsivjar please take a look

[lahsivjar]

A bit uncomfortable with the pooling but in the current code it looks good to me.