[8.x] Fix concurrent map write panic in monitoring middleware (backport #14335)

[mergify]

Motivation/summary

Fix panic due to monitoring middleware concurrent map write.

fatal error: concurrent map writes

goroutine 21913 [running]:
github.com/elastic/apm-server/internal/beater/middleware.(*monitoringMiddleware).getCounter(0xc008a53530, {0x32a4b68?, 0xc0073d1c00?})
	/home/carson/projects/apm-server/internal/beater/middleware/monitoring_middleware.go:87 +0xb8
github.com/elastic/apm-server/internal/beater/middleware.MonitoringMiddleware.(*monitoringMiddleware).Middleware.func1.1(0xc00521a000)
	/home/carson/projects/apm-server/internal/beater/middleware/monitoring_middleware.go:49 +0x59
github.com/elastic/apm-server/internal/beater/api.apmMiddleware.RecoverPanicMiddleware.func2.1(0x3617ce0?)
	/home/carson/projects/apm-server/internal/beater/middleware/recover_panic_middleware.go:60 +0x43
github.com/elastic/apm-server/internal/beater/middleware.TimeoutMiddleware.func1.1(0xc00521a000)
	/home/carson/projects/apm-server/internal/beater/middleware/timeout_middleware.go:35 +0x35
github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1(0xc00521a000)
	/home/carson/projects/apm-server/internal/beater/middleware/log_middleware.go:47 +0xaa
github.com/elastic/apm-server/internal/beater/api.NewMux.(*ContextPool).HTTPHandler.func11({0x3632730, 0xc0047620e0}, 0xc0003f88c0)
	/home/carson/projects/apm-server/internal/beater/request/context_pool.go:47 +0xec
net/http.HandlerFunc.ServeHTTP(...)
	/home/carson/apm-server-gopath/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2220
go.elastic.co/apm/module/apmhttp/v2.(*handler).ServeHTTP(0xc006b7de40, {0x3632730, 0xc0047620e0}, 0xc0003f88c0)
	/home/carson/apm-server-gopath/pkg/mod/go.elastic.co/apm/module/apmhttp/[email protected]/handler.go:74 +0xc9
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0070bd2c0, {0x3632730, 0xc0047620e0}, 0xc0003f8640)
	/home/carson/apm-server-gopath/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x2d7
github.com/elastic/gmux.ConfigureServer.(*mux).withGRPCInsecure.func3({0x3632730?, 0xc0047620e0?}, 0x2fe25c0?)
	/home/carson/apm-server-gopath/pkg/mod/github.com/elastic/[email protected]/mux.go:119 +0x105
net/http.HandlerFunc.ServeHTTP(...)
	/home/carson/apm-server-gopath/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2220
net/http.serverHandler.ServeHTTP({0x362c8e8?}, {0x3632730?, 0xc0047620e0?}, 0x6?)
	/home/carson/apm-server-gopath/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3210 +0xa2
net/http.(*conn).serve(0xc0078d7440, {0x3639618, 0xc00bcae120})
	/home/carson/apm-server-gopath/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2092 +0x610
created by net/http.(*Server).Serve in goroutine 21909
	/home/carson/apm-server-gopath/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3360 +0x485

Alternative to this, we could have populated the maps with all the possible keys, but we may risk missing some possible map keys.

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Related issues

---

This is an automatic backport of pull request #14335 done by [Mergify](https://mergify.com).

[mergify]

This pull request has not been merged yet. Could you please review and merge it @carsonip? 🙏