[BUG] unable to retrieve the complete list of server APIs: projectcalico.org/v3

[YTGhost]

What happen?

After using sealer run docker.io/sealerio/kubernetes:v1-22-15-sealerio-2 -m 172.16.158.193 -n 172.16.158.192 -p at96nW9K8C1qfXlB to deploy a cluster, I found an error about calico:

[root@iZj6cg131cph1hzu85s506Z ~]# kubectl api-resources --api-group=crd.projectcalico.org
NAME                            SHORTNAMES   APIVERSION                 NAMESPACED   KIND
bgpconfigurations                            crd.projectcalico.org/v1   false        BGPConfiguration
bgppeers                                     crd.projectcalico.org/v1   false        BGPPeer
blockaffinities                              crd.projectcalico.org/v1   false        BlockAffinity
caliconodestatuses                           crd.projectcalico.org/v1   false        CalicoNodeStatus
clusterinformations                          crd.projectcalico.org/v1   false        ClusterInformation
felixconfigurations                          crd.projectcalico.org/v1   false        FelixConfiguration
globalnetworkpolicies                        crd.projectcalico.org/v1   false        GlobalNetworkPolicy
globalnetworksets                            crd.projectcalico.org/v1   false        GlobalNetworkSet
hostendpoints                                crd.projectcalico.org/v1   false        HostEndpoint
ipamblocks                                   crd.projectcalico.org/v1   false        IPAMBlock
ipamconfigs                                  crd.projectcalico.org/v1   false        IPAMConfig
ipamhandles                                  crd.projectcalico.org/v1   false        IPAMHandle
ippools                                      crd.projectcalico.org/v1   false        IPPool
ipreservations                               crd.projectcalico.org/v1   false        IPReservation
kubecontrollersconfigurations                crd.projectcalico.org/v1   false        KubeControllersConfiguration
networkpolicies                              crd.projectcalico.org/v1   true         NetworkPolicy
networksets                                  crd.projectcalico.org/v1   true         NetworkSet
error: unable to retrieve the complete list of server APIs: projectcalico.org/v3: the server is currently unable to handle the request

Similarly, there is an associated error log in apiserver:

I0131 14:18:45.651177       1 server.go:553] external host was not specified, using 172.16.158.193
I0131 14:18:45.651660       1 server.go:161] Version: v1.22.15
I0131 14:18:46.504149       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0131 14:18:46.506452       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0131 14:18:46.506466       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0131 14:18:46.508275       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0131 14:18:46.508294       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0131 14:18:46.543802       1 genericapiserver.go:455] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0131 14:18:46.544625       1 instance.go:278] Using reconciler: lease
I0131 14:18:46.655969       1 rest.go:130] the default service ipfamily for this cluster is: IPv4
W0131 14:18:47.170075       1 genericapiserver.go:455] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0131 14:18:47.171684       1 genericapiserver.go:455] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0131 14:18:47.182522       1 genericapiserver.go:455] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0131 14:18:47.184008       1 genericapiserver.go:455] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0131 14:18:47.189409       1 genericapiserver.go:455] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0131 14:18:47.192520       1 genericapiserver.go:455] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0131 14:18:47.199251       1 genericapiserver.go:455] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0131 14:18:47.199265       1 genericapiserver.go:455] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0131 14:18:47.201096       1 genericapiserver.go:455] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0131 14:18:47.201109       1 genericapiserver.go:455] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0131 14:18:47.206423       1 genericapiserver.go:455] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0131 14:18:47.209016       1 genericapiserver.go:455] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0131 14:18:47.219293       1 genericapiserver.go:455] Skipping API apps/v1beta2 because it has no resources.
W0131 14:18:47.219313       1 genericapiserver.go:455] Skipping API apps/v1beta1 because it has no resources.
W0131 14:18:47.224242       1 genericapiserver.go:455] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
I0131 14:18:47.228993       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0131 14:18:47.229006       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0131 14:18:47.248043       1 genericapiserver.go:455] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
I0131 14:18:48.141718       1 dynamic_cafile_content.go:155] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0131 14:18:48.141752       1 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0131 14:18:48.141906       1 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0131 14:18:48.142190       1 secure_serving.go:266] Serving securely on [::]:6443
I0131 14:18:48.142258       1 apf_controller.go:312] Starting API Priority and Fairness config controller
I0131 14:18:48.142261       1 controller.go:83] Starting OpenAPI AggregationController
I0131 14:18:48.142292       1 available_controller.go:491] Starting AvailableConditionController
I0131 14:18:48.142308       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0131 14:18:48.142314       1 dynamic_serving_content.go:129] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0131 14:18:48.142310       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0131 14:18:48.142359       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0131 14:18:48.142371       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0131 14:18:48.142671       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0131 14:18:48.142678       1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I0131 14:18:48.142705       1 autoregister_controller.go:141] Starting autoregister controller
I0131 14:18:48.142707       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0131 14:18:48.142742       1 controller.go:85] Starting OpenAPI controller
I0131 14:18:48.142777       1 naming_controller.go:291] Starting NamingConditionController
I0131 14:18:48.142799       1 establishing_controller.go:76] Starting EstablishingController
I0131 14:18:48.142818       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0131 14:18:48.142833       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0131 14:18:48.142851       1 crd_finalizer.go:266] Starting CRDFinalizer
I0131 14:18:48.142709       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0131 14:18:48.145064       1 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0131 14:18:48.145105       1 dynamic_cafile_content.go:155] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0131 14:18:48.145441       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0131 14:18:48.145451       1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
E0131 14:18:48.151970       1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.16.158.193, ResourceVersion: 0, AdditionalErrorMsg:
E0131 14:18:48.164051       1 fieldmanager.go:197] "[SHOULD NOT HAPPEN] failed to update managedFields" err="failed to convert new object (/v1, Kind=Node) to smd typed: .status.addresses: duplicate entries for key [type=\"InternalIP\"]" VersionKind="/, Kind="
I0131 14:18:48.192316       1 controller.go:611] quota admission added evaluator for: namespaces
I0131 14:18:48.204708       1 shared_informer.go:247] Caches are synced for node_authorizer
I0131 14:18:48.242387       1 apf_controller.go:317] Running API Priority and Fairness config worker
I0131 14:18:48.242397       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0131 14:18:48.242413       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0131 14:18:48.242846       1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
I0131 14:18:48.242891       1 cache.go:39] Caches are synced for autoregister controller
I0131 14:18:48.245480       1 shared_informer.go:247] Caches are synced for crd-autoregister
I0131 14:18:49.141754       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0131 14:18:49.146578       1 storage_scheduling.go:132] created PriorityClass system-node-critical with value 2000001000
I0131 14:18:49.149274       1 storage_scheduling.go:132] created PriorityClass system-cluster-critical with value 2000000000
I0131 14:18:49.149289       1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I0131 14:18:49.434663       1 controller.go:611] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0131 14:18:49.460515       1 controller.go:611] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0131 14:18:49.504707       1 lease.go:233] Resetting endpoints for master service "kubernetes" to [172.16.158.193]
I0131 14:18:49.505655       1 controller.go:611] quota admission added evaluator for: endpoints
I0131 14:18:49.508431       1 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io
E0131 14:18:50.808511       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"client disconnected"}: client disconnected
E0131 14:18:50.808661       1 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout
E0131 14:18:50.810588       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0131 14:18:50.811694       1 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout
E0131 14:18:50.812873       1 timeout.go:139] post-timeout activity - time-elapsed: 4.214968ms, POST "/api/v1/namespaces/default/events" result: <nil>
I0131 14:18:50.977819       1 controller.go:611] quota admission added evaluator for: serviceaccounts
I0131 14:18:50.991888       1 controller.go:611] quota admission added evaluator for: deployments.apps
I0131 14:18:51.110855       1 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io
E0131 14:18:51.222531       1 fieldmanager.go:197] "[SHOULD NOT HAPPEN] failed to update managedFields" err="failed to convert new object (/v1, Kind=Node) to smd typed: .status.addresses: duplicate entries for key [type=\"InternalIP\"]" VersionKind="/, Kind="
I0131 14:18:51.587287       1 controller.go:611] quota admission added evaluator for: daemonsets.apps
I0131 14:19:05.343699       1 controller.go:611] quota admission added evaluator for: replicasets.apps
I0131 14:19:05.592147       1 controller.go:611] quota admission added evaluator for: controllerrevisions.apps
I0131 14:19:06.830011       1 controller.go:611] quota admission added evaluator for: events.events.k8s.io
E0131 14:19:13.112516       1 fieldmanager.go:197] "[SHOULD NOT HAPPEN] failed to update managedFields" err="failed to convert new object (/v1, Kind=Node) to smd typed: .status.addresses: duplicate entries for key [type=\"InternalIP\"]" VersionKind="/, Kind="
I0131 14:19:21.360821       1 controller.go:611] quota admission added evaluator for: poddisruptionbudgets.policy
I0131 14:19:47.483165       1 trace.go:205] Trace[78131753]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-izj6cg131cph1hzu85s506z/log,user-agent:k9s/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:721bf71c-0492-42c6-89b0-3744fcb420d6,client:172.16.158.193,accept:application/json, */*,protocol:HTTP/2.0 (31-Jan-2023 14:19:35.328) (total time: 12155ms):
Trace[78131753]: ---"Transformed response object" 12153ms (14:19:47.483)
Trace[78131753]: [12.155062215s] [12.155062215s] END
I0131 14:19:52.289120       1 controller.go:611] quota admission added evaluator for: networkpolicies.networking.k8s.io
W0131 14:19:53.194548       1 handler_proxy.go:107] no RequestInfo found in the context
E0131 14:19:53.194637       1 controller.go:116] loading OpenAPI spec for "v3.projectcalico.org" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0131 14:19:53.194649       1 controller.go:129] OpenAPI AggregationController: action for item v3.projectcalico.org: Rate Limited Requeue.
E0131 14:20:07.588134       1 available_controller.go:524] v3.projectcalico.org failed with: failing or missing response from https://100.105.175.129:5443/apis/projectcalico.org/v3: Get "https://100.105.175.129:5443/apis/projectcalico.org/v3": context deadline exceeded
I0131 14:20:27.022724       1 trace.go:205] Trace[1656319656]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-izj6cg131cph1hzu85s506z/log,user-agent:k9s/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:01fd2a42-e813-479f-a23b-f21c11da39e9,client:172.16.158.193,accept:application/json, */*,protocol:HTTP/2.0 (31-Jan-2023 14:19:50.503) (total time: 36518ms):
Trace[1656319656]: ---"Transformed response object" 36517ms (14:20:27.022)
Trace[1656319656]: [36.518950845s] [36.518950845s] END
E0131 14:20:38.589462       1 controller.go:116] loading OpenAPI spec for "v3.projectcalico.org" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp 100.105.175.129:5443: i/o timeout
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0131 14:20:38.589478       1 controller.go:129] OpenAPI AggregationController: action for item v3.projectcalico.org: Rate Limited Requeue.
I0131 14:21:24.250742       1 trace.go:205] Trace[1730059489]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-izj6cg131cph1hzu85s506z/log,user-agent:k9s/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:1f8b5417-b2c0-43a0-b95d-e132f50556b7,client:172.16.158.193,accept:application/json, */*,protocol:HTTP/2.0 (31-Jan-2023 14:20:56.400) (total time: 27850ms):
Trace[1730059489]: ---"Transformed response object" 27849ms (14:21:24.250)
Trace[1730059489]: [27.850608962s] [27.850608962s] END
E0131 14:22:53.188129       1 available_controller.go:524] v3.projectcalico.org failed with: failing or missing response from https://100.105.175.129:5443/apis/projectcalico.org/v3: Get "https://100.105.175.129:5443/apis/projectcalico.org/v3": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0131 14:23:08.754996       1 controller.go:116] loading OpenAPI spec for "v3.projectcalico.org" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp 100.105.175.129:5443: i/o timeout
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0131 14:23:08.755013       1 controller.go:129] OpenAPI AggregationController: action for item v3.projectcalico.org: Rate Limited Requeue.
E0131 14:24:19.187201       1 controller.go:116] loading OpenAPI spec for "v3.projectcalico.org" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp 100.105.175.129:5443: i/o timeout
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0131 14:24:19.187216       1 controller.go:129] OpenAPI AggregationController: action for item v3.projectcalico.org: Rate Limited Requeue.
E0131 14:26:49.193527       1 controller.go:116] loading OpenAPI spec for "v3.projectcalico.org" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp 100.105.175.129:5443: i/o timeout
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0131 14:26:49.193542       1 controller.go:129] OpenAPI AggregationController: action for item v3.projectcalico.org: Rate Limited Requeue.

Relevant log output?

[root@iZj6cg131cph1hzu85s506Z ~]# kubectl get apiservice
NAME                                   SERVICE                       AVAILABLE   AGE
v1.                                    Local                         True        15m
v1.admissionregistration.k8s.io        Local                         True        15m
v1.apiextensions.k8s.io                Local                         True        15m
v1.apps                                Local                         True        15m
v1.authentication.k8s.io               Local                         True        15m
v1.authorization.k8s.io                Local                         True        15m
v1.autoscaling                         Local                         True        15m
v1.batch                               Local                         True        15m
v1.certificates.k8s.io                 Local                         True        15m
v1.coordination.k8s.io                 Local                         True        15m
v1.crd.projectcalico.org               Local                         True        14m
v1.discovery.k8s.io                    Local                         True        15m
v1.events.k8s.io                       Local                         True        15m
v1.networking.k8s.io                   Local                         True        15m
v1.node.k8s.io                         Local                         True        15m
v1.operator.tigera.io                  Local                         True        14m
v1.policy                              Local                         True        15m
v1.rbac.authorization.k8s.io           Local                         True        15m
v1.scheduling.k8s.io                   Local                         True        15m
v1.storage.k8s.io                      Local                         True        15m
v1beta1.batch                          Local                         True        15m
v1beta1.discovery.k8s.io               Local                         True        15m
v1beta1.events.k8s.io                  Local                         True        15m
v1beta1.flowcontrol.apiserver.k8s.io   Local                         True        15m
v1beta1.node.k8s.io                    Local                         True        15m
v1beta1.policy                         Local                         True        15m
v1beta1.storage.k8s.io                 Local                         True        15m
v2beta1.autoscaling                    Local                         True        15m
v2beta2.autoscaling                    Local                         True        15m
v3.projectcalico.org                   calico-apiserver/calico-api   True        14m

No response

What you expected to happen?

no error

How to reproduce it (as minimally and precisely as possible)?

sealer run docker.io/sealerio/kubernetes:v1-22-15-sealerio-2 -m 172.16.158.193 -n 172.16.158.192 -p xxxxxxxxx

Anything else we need to know?

No response

What is the version of Sealer you using?

{"gitVersion":"v0.9.1","gitCommit":"9934e11","buildDate":"2023-01-17 11:30:38","goVersion":"go1.17.13","compiler":"gc","platform":"linux/amd64"}

What is your OS environment?

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

What is the Kernel version?

Linux iZj6cg131cph1hzu85s506Z 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Other environment you want to tell us?

• Cloud provider or hardware configuration:
• Install tools:
• Others:

[starnop]

After testing, it is found that the calico network is abnormal because other network plug-ins have been installed in the environment and not cleaned up.

It's not a matter of cluster image provided by Sealer community, I'm going to close this issue