Failed ReopenContainerLog from runtime interface

[OleksiiKhanin]

There is an error when I use cri-dockerd as container-runtime.

Kubelet tries to reopen log file, but get the error from container runtime.

E0209 21:28:39.416074 1071258 remote_runtime.go:1134] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID=....

E0209 21:28:39.981058 1071258 container_log_manager.go:244] "Container log doesn't exist, reopen container log failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID=.....

It stuck when I try to run kubernetes metrics-server.
When I run the same helm chart with kubelet build in dockerhim it works fine.

kubectl version:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25",  BuildDate:"2022-01-28T07:59:30Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

OS:  5.10.93-flatcar

Also, I find the same error as mine here rancher/rke#2716

Do you have any ideas how to fix this, thanks?

[chauncyAAA]

core/logs.go

// ReopenContainerLog reopens the container log file.
func (ds *dockerService) ReopenContainerLog(
_ context.Context,
_ *runtimeapi.ReopenContainerLogRequest,
) (*runtimeapi.ReopenContainerLogResponse, error) {
return nil, fmt.Errorf("docker does not support reopening container log files")
}

[nickvth]

Why is this closed? Still have this on RKE clusters with k8s 1.24.8 and cri-dockerd enabled
Or is there a fix/workaround?

[evol262]

@nickvth because a fix from Rancher was merged in as part of #38. I'll check the metrics server for 1.24.8 to see if they changed what they're expecting here, because Docker has never been able to re-open container logs, including in the in-tree dockershim

[nickvth]

@evol262 fixed when setting container-log-max-size to the same size as docker daemon max-size log opts.