Add sourcemap periodic refresh tracing

[dmathieu]

Motivation/summary

Sourcemap periodic refreshes are making external calls to ES. Those calls produce spans, but they can't generate any data, as there is no parent transaction.
This creates a transaction for each sync run, so we can observe them.

Checklist

• Update CHANGELOG.asciidoc
• Update package changelog.yml (only if changes to apmpackage have been made)
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Outside of unit tests, I have been testing these changes by running apm-server with tilt and rum/self instrumentation enabled.

I'm not 100% sure about the initialSearch and scrollSearch spans, since they do nothing but wrap the ES call. They also help in figuring out which method makes those calls though.

Related issues

See #1014.

[mergify]

This pull request does not have a backport label. Could you fix it @dmathieu? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-07-03T08:30:49.320+0000

• Duration: 6 min 12 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[apmmachine]

📚 Go benchmark report

Diff with the main branch

goos: linux
goarch: amd64
pkg: github.com/elastic/apm-server/internal/agentcfg
cpu: 12th Gen Intel(R) Core(TM) i5-12500
                                  │ build/main/bench.out │             bench.out             │
                                  │        sec/op        │   sec/op     vs base              │
geomean                                      66.65n        66.84n       +0.28%

                                  │ build/main/bench.out │             bench.out              │
                                  │         B/op         │    B/op     vs base                │
geomean                                                ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                  │ build/main/bench.out │             bench.out              │
                                  │      allocs/op       │ allocs/op   vs base                │
geomean                                                ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/elastic/apm-server/internal/beater/request
                                             │ build/main/bench.out │             bench.out              │
                                             │        sec/op        │    sec/op     vs base              │
ContextResetContentEncoding/empty-12                   130.7n ±  1%   131.4n ±  1%  +0.54% (p=0.039 n=6)
geomean                                                910.0n         935.7n        +2.83%

                                             │ build/main/bench.out │              bench.out               │
                                             │         B/op         │     B/op      vs base                │
geomean                                                           ²                 +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                             │ build/main/bench.out │             bench.out              │
                                             │      allocs/op       │ allocs/op   vs base                │
geomean                                                           ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/elastic/apm-server/internal/publish
             │ build/main/bench.out │          bench.out          │
             │        sec/op        │   sec/op    vs base         │

             │ build/main/bench.out │           bench.out            │
             │         B/op         │     B/op       vs base         │

             │ build/main/bench.out │           bench.out           │
             │      allocs/op       │  allocs/op    vs base         │

pkg: github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics
                 │ build/main/bench.out │             bench.out             │
                 │        sec/op        │   sec/op     vs base              │
AggregateSpan-12            881.0n ± 2%   913.8n ± 4%  +3.71% (p=0.002 n=6)

                 │ build/main/bench.out │            bench.out            │
                 │         B/op         │     B/op      vs base           │
¹ all samples are equal

                 │ build/main/bench.out │           bench.out           │
                 │      allocs/op       │ allocs/op   vs base           │
¹ all samples are equal

pkg: github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics
                        │ build/main/bench.out │          bench.out           │
                        │        sec/op        │   sec/op     vs base         │

                        │ build/main/bench.out │           bench.out           │
                        │         B/op         │    B/op     vs base           │
¹ all samples are equal

                        │ build/main/bench.out │           bench.out           │
                        │      allocs/op       │ allocs/op   vs base           │
¹ all samples are equal

pkg: github.com/elastic/apm-server/x-pack/apm-server/sampling
               │ build/main/bench.out │             bench.out              │
               │        sec/op        │    sec/op     vs base              │
geomean                  453.6n         468.1n        +3.20%

               │ build/main/bench.out │              bench.out               │
               │         B/op         │     B/op      vs base                │
geomean                    381.1          380.6       -0.13%
¹ all samples are equal

               │ build/main/bench.out │             bench.out              │
               │      allocs/op       │ allocs/op   vs base                │
geomean                    4.123        4.123       +0.00%
¹ all samples are equal

pkg: github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage
                                            │ build/main/bench.out │               bench.out               │
                                            │        sec/op        │     sec/op      vs base               │
WriteTransaction/json_codec-12                       1.809µ ±  19%    1.344µ ±  62%  -25.68% (p=0.015 n=6)
ReadEvents/json_codec_big_tx/399_events-12           1.228m ±   3%    1.263m ±   2%   +2.87% (p=0.041 n=6)
ReadEvents/json_codec_big_tx/1000_events-12          3.068m ±   2%    2.989m ±   4%   -2.57% (p=0.026 n=6)
ReadEvents/nop_codec/10_events-12                    10.22µ ±  21%    14.35µ ±  28%  +40.44% (p=0.015 n=6)
geomean                                              15.16µ           15.17µ          +0.06%

                                            │ build/main/bench.out │               bench.out               │
                                            │         B/op         │     B/op       vs base                │
ReadEvents/json_codec_big_tx/10_events-12            42.31Ki ±  0%   42.33Ki ±  0%  +0.04% (p=0.041 n=6)
ReadEvents/nop_codec_big_tx/199_events-12            122.0Ki ±  1%   122.8Ki ±  0%  +0.63% (p=0.015 n=6)
geomean                                              11.33Ki         11.38Ki        +0.49%
¹ all samples are equal

                                            │ build/main/bench.out │              bench.out              │
                                            │      allocs/op       │  allocs/op   vs base                │
geomean                                                 127.4         127.4       -0.00%
¹ all samples are equal

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

[dmathieu]

System tests failed once du to a flaky test. They passed on a second run, but somehow, the System Tests Results is still marked as failed. And rerunning it doesn't change anything.

[carsonip]

Would be great to mention this is for instrumentation only.

[dmathieu]

How about now?

[axw]

Verified with 8.10.0-BC1. Running apm-server under Tilt, I configured instrumentation.enabled: true and apm-server.rum.enabled: true and observed the "MetadataESFetcher.sync" traces being captured:

I think the transaction name could mention sourcemaps somewhere, but this is good enough for now.