[AllBeats]Kubernetes: Move away from extensions/v1beta1

[odacremolbap]

Describe the enhancement:
extensions/v1beta1 have been deprecated as a group for DaemonSets and ReplicaSets among other objects.

Some of our deployments use that ApiGroup, which will make it fail at kubernetes 1.16
Moving to apps/v1 will make deployments fail at kubernetes 1.8 and below.

As per statistics at the ECK project, there are no users for them using 1.8, but we can expect people using latest minikube images targeting 1.16.

This issue affects mainly documentation, but there is also an import from the kubernetes API project using extensions/v1beta1. While it seems that it should still work, new releases of the library will probably remove the package.

[nerddelphi]

I'm using Metricbeat 7.4.0 and I'm getting these issues:

E1002 17:35:07.668403 1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1.StatefulSet, but watch event object had type *v1.ReplicaSet │ │ E1002 17:35:07.690845 1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1.StatefulSet, but watch event object had type *v1.ReplicaSet │ │ E1002 17:35:07.691428 1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1beta1.ReplicaSet, but watch event object had type *v1.ReplicaSet │ │ E1002 17:35:11.492552 1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1beta1.ReplicaSet, but watch event object had type *v1.ReplicaSet │ │ E1002 17:35:11.493101 1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1.StatefulSet, but watch event object had type *v1.ReplicaSet

and
E1002 17:05:43.831446 1 reflector.go:125] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: Failed to list *v1.StatefulSet: replicasets.apps is forbidden: User "system:serviceaccount:elastic-stack:metricbeat" cannot list resource "replicasets" in API group "apps" at the cluster scope

[odacremolbap]

@nerddelphi sorry to hear that.

Are you using kubernetes v1.16?
If possible I would suggest to downgrade to v1.15and test.

[nerddelphi]

@odacremolbap I'm using kubernetes 1.13.10 and getting that error.

[odacremolbap]

Can you provide the configuration you are using so I can reproduce?

I would prefer an issue at discuss until we confirm that the issue you are hitting is related to the feature at this ticket.

[nerddelphi]

@odacremolbap issue opened at discuss.

Thanks.

[odacremolbap]

Thanks a lot @nerddelphi
That's a typo at our code that was trying to retrieve Replicasets at the watcher when a Statefulset was received.

Expect the fix for 7.4.1

(This thread will kept open after fixing because the original issue here was removing references to extensions/v1beta1)

[nerddelphi]

No prob, @odacremolbap .

After the release, I'll test and back to you.

Thank you.

[odacremolbap]

Hi @nerddelphi

The fix was released at 7.5.0, but we have an early report #13905 (comment) from @tkzv that indicates that this is still an issue.

We would appreciate you testing your scenario with 7.5.0, I will do so in the next days (currently OOO)

[nerddelphi]

Hi @odacremolbap . The issue still exists.

E1212 16:56:40.196944 1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1beta1.ReplicaSet, but watch event object had type *v1.ReplicaSet │
│ E1212 16:56:53.675078 1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1beta1.ReplicaSet, but watch event object had type *v1.ReplicaSet

[tkzv]

Here are the log and configuration files.
metricbeat-deployment.log
config.zip
The only information displayed in Kibana's "[Metricbeat Kubernetes] Overview ECS" are the numbers of nodes, deployments and pods. Network, CPU and memory are absent.

[bitva77]

7.5.1 here and I'm seeing this issue as well in the Metricbeat Deployment pod

E0117 00:51:54.251362       1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1beta1.ReplicaSet, but watch event object had type *v1.ReplicaSet

We're running Kubernetes 1.15.2

[PhaedrusTheGreek]

I'm seeing the following in 7.5.1:

E0117 18:29:20.397991       1 reflector.go:125] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: Failed to list *v1beta1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:metricbeat" cannot list resource "replicasets" in API group "apps" at the cluster scope

[bitva77]

@PhaedrusTheGreek I was getting the same as well. The following ClusterRole seems to fix it:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    k8s-app: metricbeat
rules:
- apiGroups: [""]
  resources:
  - nodes
  - namespaces
  - events
  - pods
  - nodes/metrics
  - nodes/stats
  - deployments
  - nodes
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs: ["get", "list", "watch"]

[nerddelphi]

The only one component of my Stack using a previous working version is the Metricbeat: 7.3.2.

I've opened a new issue to treat that issue: #15556

[nerddelphi]

I've got the same issue in 7.5.2 as well.

[tkzv]

@bitva77, didn't work for me.
"Network out by node", "Network in by node", "Top CPU intensive pods" and "Top memory intensive pods" still show "No data to display for the selected metrics".

[marcossousaskynz]

I'm seeing this issue on metricbeat 7.5.2 as well

[wajika]

This problem still exists in version 7.8.1