feat: consume APM configuration over the elastic-agent control protocol

[kruskall]

Motivation/summary

Consume APM configuration over the elastic-agent control protocol. Pin elastic-agent-client version to 7.6.0 until beats is updated.

Create a new instrumentation package to retrieve the tracer dynamically. It might need some changes in go-docappender and other packages to allow updating the tracer.

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Testing is a nightmare and can't be easily performed. We can't run elastic-agent locally and the ea-managed apm-server system test is broken.
Current approach is to start the ea-managed smoke test, manually install elastic-agent, upload apm-server binary to replace the apm-server component and restart elastic-agent.

Related issues

[mergify]

This pull request does not have a backport label. Could you fix it @kruskall? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.

NOTE: backport-skip has been added to this pull request.

[cmacknz]

Testing is a nightmare and can't be easily performed. We can't run elastic-agent locally

Best solution to this is a VM, I like https://multipass.run/ especially if you are a Mac. Makes dealing with Ubuntu VMs as easy as containers.

[cmacknz]

In libbeat we do this in https://github.com/elastic/beats/blob/main/x-pack/libbeat/management/managerV2.go#L176-L183

		// Normal Elastic-Agent-Client initialisation
		agentClient, _, err = client.NewV2FromReader(os.Stdin, client.VersionInfo{
			Name:    "beat-v2-client",
			Version: version.GetDefaultVersion(),
			Meta: map[string]string{
				"commit":     version.Commit(),
				"build_time": version.BuildTime().String(),
			},

This is called during initialization from https://github.com/elastic/beats/blob/c5e79a25d05d5bdfa9da4d187fe89523faa42afc/libbeat/cmd/instance/beat.go#L837-L841

	// initialize config manager
	b.Manager, err = management.NewManager(b.Config.Management, reload.RegisterV2)
	if err != nil {
		return err
	}

In APM Server I see this is already called in

apm-server/internal/beatcmd/beat.go

Lines 152 to 157 in 062c286

	// Initialize central config manager.
	manager, err := management.NewManager(b.Config.Management, reload.RegisterV2)
	if err != nil {
	return err
	}
	b.Manager = manager

So what I suspect is happening is that you are reading an empty proto message from stdin because the one the agent wrote is already consumed. This is causing the Services field to be nil failing the check in https://github.com/elastic/elastic-agent-client/blob/17d2b257ff148cd86de88252359bef0f28df9ab8/pkg/client/reader.go#L86

The io.ReadAll must just be reading EOF if you aren't getting an error back, which seems weird but possible.

Regardless of the actual mechanism causing this to fail you are trying to initialize the client a second time which is likely the source of your problems.

[kruskall]

Thank you! 🙇

[v1v]

run docs-build

[kruskall]

This requires changes in beats, I'll close this for now