Improve ECS support in operator logs

[barkbay]

Following up #2002, #2457 added some support for ECS but some improvements can be done:

1. Should we consider a "reconciliation loop" as an event?

• event.sequence looks like our iteration field (number of times a controller has run its Reconcile method)
• event.duration looks like our current took
• event.type could be [ change| creation | deletion ]
• event.dataset could help to separate the logs from the different controller (elasticsearch-controller, kibana-controller...)

2. The presence of some fields is inconsistent

It is worth to note that it is not easy to be consistent re. the key/values added to the log.
The most problematic case is when an error is managed by the controller itself (i.e. not by our code) :

if result, err := c.Do.Reconcile(req); err != nil {
   c.Queue.AddRateLimited(req)
   log.Error(err, "Reconciler error", "controller", c.Name, "request", req)
   [...]
   return false
}

This would produce:

{
	"message": "Reconciler error",
	"log.logger": "controller-runtime.controller",
	"log.level": "error",
	"controller": "kibana-association-controller",
	"error.message": "An error message about kibanas.kibana.k8s.elastic.co \"kb-apm-sample\": something bad happened",
	"error.stack_trace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88",
	"request": "namespace1/kb-apm-sample"
}

In the above example if a user is filtering on namespace: "namespace1" and kibana_name: "kb-apm-sample" then the error will not be displayed.
We could propose to the controller-runtime project to replace "request": "namespace1/kb-apm-sample" by something like:

log.Error(err, "Reconciler error", "controller", c.Name, "namespace", req.Namespace, "name", req.name)

(we would still have a problem with the "name" field but this could be revisited in our code)
Or add a processor or our own encoder to transform the output.

We also have some inconsistencies in our code, mostly in the "common" packages, for example when a certificate is generated:

cloud-on-k8s/pkg/controller/common/certificates/http/reconcile.go

Line 145 in b1355ce

log.Info("Creating HTTP internal certificate secret", "namespace", secret.Namespace, "secret_name", secret.Name)

I think it would be nice to add a reference to the owner in order to link this log to the parent resource.

3. Once Add basic APM agent instrumentation #2462 is merged we should add some correlation fields (see Add basic APM agent instrumentation #2462 (review))