Support loadbalancerClass #100

lubronzhan · 2024-01-31T07:33:42Z

Summary

Now if kube-vip-cloud-provider set env variable KUBEVIP_ENABLE_LOADBALANCERCLASS: true, then it will only reconcile service that has : spec.loadbalancerClass: kube-vip.io/kube-vip-class

To implement that, I added a second service controller, since cloud-provider will ignore service type LB with loadbalancerclass set. Then if KUBEVIP_ENABLE_LOADBALANCERCLASS is set to true, it will only start the newly added custom service controller. If not set, it will not start this newly added custom service controller.

In the future, we can just get rid of the cloud-provider controller completely, and just use this new custom service controller to reconcile all service type LB.

Manual Test

Without loadbalancerclass

Not adding loadbalancerclass, the log is still the same as previously. log when creating lb svc without loadbalancerclass,

## without lbclass
apiVersion: v1
kind: Service
metadata:
  name: example-service
spec:
  selector:
    app: example
  ports:
    - port: 8765
      targetPort: 9376
  type: LoadBalancer

no log when creating lb svc loadbalancerclass

apiVersion: v1
kind: Service
metadata:
  name: example-service-loadbalancerclass
spec:
  selector:
    app: example
  ports:
    - port: 8765
      targetPort: 9376
  type: LoadBalancer
  loadBalancerClass: kube-vip.io/kube-vip-class

  kube-vip-cloud-provider-ccbc46867-jqtml -f
      kube-vip-cloud-provider-7c59d78b6-vl4rq -f
I0204 18:49:58.769622       1 serving.go:380] Generated self-signed cert in-memory
I0204 18:49:59.771877       1 serving.go:380] Generated self-signed cert in-memory
W0204 18:49:59.772122       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0204 18:50:01.459349       1 provider.go:84] staring with loadbalancerClass set to: false
I0204 18:50:01.459829       1 provider.go:86] Watching configMap for pool config with name: 'kubevip', namespace: 'kube-system'
I0204 18:50:01.460825       1 controllermanager.go:169] Version: v0.0.2
I0204 18:50:01.476161       1 secure_serving.go:213] Serving securely on [::]:10258
I0204 18:50:01.476179       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0204 18:50:01.481931       1 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-vip-cloud-controller...
I0204 18:50:20.389647       1 leaderelection.go:260] successfully acquired lease kube-system/kube-vip-cloud-controller
I0204 18:50:20.393518       1 event.go:376] "Event occurred" object="kube-system/kube-vip-cloud-controller" fieldPath="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="kube-vip-cloud-provider-7c59d78b6-vl4rq_f4749941-1db7-46a7-a4d0-df3ba7a1f032 became leader"
I0204 18:50:20.400511       1 provider.go:123] Initing Kube-vip Cloud Provider
I0204 18:50:20.408159       1 controllermanager.go:338] Started "service-lb-controller"
I0204 18:50:20.410145       1 controller.go:231] Starting service controller
I0204 18:50:20.416155       1 shared_informer.go:311] Waiting for caches to sync for service
I0204 18:50:20.530314       1 shared_informer.go:318] Caches are synced for service
I0204 18:50:23.526131       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0204 18:50:23.547117       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="UnAvailableLoadBalancer" message="There are no available nodes for LoadBalancer"
I0204 18:50:23.547153       1 loadBalancer.go:92] syncing service 'example-service' (6e4692e0-1655-4c4f-a66b-b2f33edc1392)
I0204 18:50:23.566990       1 loadBalancer.go:247] no cidr config for namespace [default] exists in key [cidr-default] configmap [kubevip]
I0204 18:50:23.567154       1 loadBalancer.go:252] Taking address from [cidr-global] pool
I0204 18:50:23.575764       1 loadBalancer.go:209] Updating service [example-service], with load balancer IPAM address(es) [192.168.0.217]
I0204 18:50:23.675859       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="LoadbalancerIP" message=" -> 192.168.0.217"
I0204 18:50:23.677166       1 loadBalancer.go:92] syncing service 'example-service' (6e4692e0-1655-4c4f-a66b-b2f33edc1392)
I0204 18:50:23.677400       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"
I0204 18:50:23.677528       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0204 18:50:23.677609       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="UnAvailableLoadBalancer" message="There are no available nodes for LoadBalancer"
I0204 18:50:23.677668       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"
I0204 18:50:28.361195       1 loadBalancer.go:78] deleting service 'example-service' (6e4692e0-1655-4c4f-a66b-b2f33edc1392)
I0204 18:50:28.361523       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="DeletingLoadBalancer" message="Deleting load balancer"
I0204 18:50:28.369954       1 event.go:376] "Event occurred" object="default/example-service" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="DeletedLoadBalancer" message="Deleted load balancer"

With loadbalancerclass

env:
        - name: KUBEVIP_ENABLE_LOADBALANCERCLASS
          value: "true"

Apply a lb without lbc, it will just ignore

## without lbclass
apiVersion: v1
kind: Service
metadata:
  name: example-service
spec:
  selector:
    app: example
  ports:
    - port: 8765
      targetPort: 9376
  type: LoadBalancer

Applying a lb with correct lbc, it will reconcile

apiVersion: v1
kind: Service
metadata:
  name: example-service-loadbalancerclass
spec:
  selector:
    app: example
  ports:
    - port: 8765
      targetPort: 9376
  type: LoadBalancer
  loadBalancerClass: kube-vip.io/kube-vip-class

Applying a lb with incorrect lbc, it will ignore

apiVersion: v1
kind: Service
metadata:
  name: example-service
spec:
  selector:
    app: example
  ports:
    - port: 8765
      targetPort: 9376
  type: LoadBalancer
  loadBalancerClass: wrong

system          kube-vip-cloud-provider-bbb8944df-wqnj2 -f
I0205 04:21:15.111204       1 main.go:62] skipping default cloud-provider service controller
I0205 04:21:16.762837       1 serving.go:380] Generated self-signed cert in-memory
I0205 04:21:18.371739       1 serving.go:380] Generated self-signed cert in-memory
W0205 04:21:18.372076       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0205 04:21:20.620754       1 provider.go:78] Checking if loadbalancerClass is enabled: true
I0205 04:21:20.620890       1 provider.go:84] staring with loadbalancerClass set to: true
I0205 04:21:20.620932       1 provider.go:86] Watching configMap for pool config with name: 'kubevip', namespace: 'kube-system'
I0205 04:21:20.621811       1 controllermanager.go:169] Version: v0.0.2
I0205 04:21:20.637080       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0205 04:21:20.637268       1 secure_serving.go:213] Serving securely on [::]:10258
I0205 04:21:20.641997       1 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-vip-cloud-controller...
I0205 04:21:35.762680       1 leaderelection.go:260] successfully acquired lease kube-system/kube-vip-cloud-controller
I0205 04:21:35.767888       1 event.go:376] "Event occurred" object="kube-system/kube-vip-cloud-controller" fieldPath="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="kube-vip-cloud-provider-bbb8944df-wqnj2_c5869c09-c8ac-42a5-9337-b9389430fefd became leader"
I0205 04:21:35.778164       1 provider.go:123] Initing Kube-vip Cloud Provider
I0205 04:21:35.778853       1 provider.go:129] staring a separate service controller that only monitors service with loadbalancerClass
I0205 04:21:35.778890       1 provider.go:130] default cloud-provider service controller will ignore service with loadbalancerClass
I0205 04:21:35.781245       1 shared_informer.go:311] Waiting for caches to sync for service
I0205 04:21:35.883157       1 shared_informer.go:318] Caches are synced for service
I0205 04:21:35.884130       1 loadbalancerclass.go:180] Reconcile service default/example-service-loadbalancerclass, since loadbalancerClass match
I0205 04:21:35.884578       1 loadBalancer.go:92] syncing service 'example-service-loadbalancerclass' (e71b22f7-1caf-4f59-a7f4-2020c59d033f)
I0205 04:21:35.884751       1 loadbalancerclass.go:192] Finished processing service default/example-service-loadbalancerclass (501.125µs)
I0205 04:21:35.884703       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"e71b22f7-1caf-4f59-a7f4-2020c59d033f", APIVersion:"v1", ResourceVersion:"85609", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0205 04:21:35.886290       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"e71b22f7-1caf-4f59-a7f4-2020c59d033f", APIVersion:"v1", ResourceVersion:"85609", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
I0205 04:24:40.061478       1 loadbalancerclass.go:180] Reconcile service default/example-service-loadbalancerclass, since loadbalancerClass match
I0205 04:24:40.063078       1 loadbalancerclass.go:247] Removing finalizer from service default/example-service-loadbalancerclass
I0205 04:24:40.098703       1 loadbalancerclass.go:192] Finished processing service default/example-service-loadbalancerclass (36.906958ms)
I0205 04:24:40.098893       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"e71b22f7-1caf-4f59-a7f4-2020c59d033f", APIVersion:"v1", ResourceVersion:"112355", FieldPath:""}): type: 'Normal' reason: 'LoadBalancerDeleted' Deleted load balancer
I0205 04:25:27.305409       1 loadbalancerclass.go:180] Reconcile service default/example-service-loadbalancerclass, since loadbalancerClass match
I0205 04:25:27.307576       1 loadbalancerclass.go:232] Adding finalizer to service default/example-service-loadbalancerclass
I0205 04:25:27.312225       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"befe28af-778c-4e7e-87f4-a70369f22a72", APIVersion:"v1", ResourceVersion:"112460", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0205 04:25:27.317208       1 loadBalancer.go:92] syncing service 'example-service-loadbalancerclass' (befe28af-778c-4e7e-87f4-a70369f22a72)
I0205 04:25:27.320471       1 loadBalancer.go:247] no cidr config for namespace [default] exists in key [cidr-default] configmap [kubevip]
I0205 04:25:27.320562       1 loadBalancer.go:252] Taking address from [cidr-global] pool
I0205 04:25:27.328126       1 loadBalancer.go:209] Updating service [example-service-loadbalancerclass], with load balancer IPAM address(es) [192.168.0.217]
I0205 04:25:27.333978       1 loadbalancerclass.go:192] Finished processing service default/example-service-loadbalancerclass (28.282167ms)
I0205 04:25:27.334578       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"befe28af-778c-4e7e-87f4-a70369f22a72", APIVersion:"v1", ResourceVersion:"112460", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
I0205 04:25:27.336118       1 loadbalancerclass.go:180] Reconcile service default/example-service-loadbalancerclass, since loadbalancerClass match
I0205 04:25:27.336174       1 loadBalancer.go:92] syncing service 'example-service-loadbalancerclass' (befe28af-778c-4e7e-87f4-a70369f22a72)
I0205 04:25:27.336205       1 loadbalancerclass.go:192] Finished processing service default/example-service-loadbalancerclass (56.25µs)
I0205 04:25:27.336255       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"befe28af-778c-4e7e-87f4-a70369f22a72", APIVersion:"v1", ResourceVersion:"112464", FieldPath:""}): type: 'Normal' reason: 'LoadbalancerIP'  -> 192.168.0.217
I0205 04:25:27.336328       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"befe28af-778c-4e7e-87f4-a70369f22a72", APIVersion:"v1", ResourceVersion:"112464", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0205 04:25:27.336365       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"befe28af-778c-4e7e-87f4-a70369f22a72", APIVersion:"v1", ResourceVersion:"112464", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
I0205 04:25:30.834572       1 loadbalancerclass.go:180] Reconcile service default/example-service-loadbalancerclass, since loadbalancerClass match
I0205 04:25:30.834977       1 loadbalancerclass.go:247] Removing finalizer from service default/example-service-loadbalancerclass
I0205 04:25:30.865099       1 loadbalancerclass.go:192] Finished processing service default/example-service-loadbalancerclass (30.2935ms)
I0205 04:25:30.865326       1 event.go:364] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"example-service-loadbalancerclass", UID:"befe28af-778c-4e7e-87f4-a70369f22a72", APIVersion:"v1", ResourceVersion:"112474", FieldPath:""}): type: 'Normal' reason: 'LoadBalancerDeleted' Deleted load balancer

Signed-off-by: lubronzhan <[email protected]>

thebsdbox · 2024-02-06T08:45:12Z

Is this ready to roll?

lubronzhan · 2024-02-06T21:20:54Z

Yes, I tested with kube-vip setting lbclassOnly as well. Just make sure you are ok with the change

kubo@bzGUizLoQEdDH:~$ kubectl get svc my-nginx-1 -oyaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    kube-vip.io/loadbalancerIPs: 10.83.33.234
    kube-vip.io/vipHost: wl-antrea-controlplane-7lb6r-ztv4w
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"my-nginx-1","namespace":"default"},"spec":{"externalTrafficPolicy":"Cluster","internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","loadBalancerClass":"kube-vip.io/kube-vip-class","ports":[{"port":80,"protocol":"TCP","targetPort":80}],"selector":{"run":"my-nginx-1"},"type":"LoadBalancer"},"status":{"loadBalancer":{}}}
  creationTimestamp: "2024-02-06T21:19:33Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    implementation: kube-vip
  name: my-nginx-1
  namespace: default
  resourceVersion: "14703"
  uid: aabbae45-f216-4231-9ff7-0210e8019fab
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 100.70.35.27
  clusterIPs:
  - 100.70.35.27
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  loadBalancerClass: kube-vip.io/kube-vip-class
  loadBalancerIP: 10.83.33.234
  ports:
  - nodePort: 30991
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: my-nginx-1
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 10.83.33.234

lubronzhan force-pushed the topic/lubron/fix-61 branch from 29c2b6c to 43bd8e8 Compare January 31, 2024 07:43

lubronzhan added 5 commits February 3, 2024 19:15

Inital controller code

495cea7

Signed-off-by: lubronzhan <[email protected]>

Add a service controller to handle loadbalancerclass case

4f85450

Signed-off-by: lubronzhan <[email protected]>

Unit test

dfdb8e4

Signed-off-by: lubronzhan <[email protected]>

Fix logic to init

fd4fff8

Signed-off-by: lubronzhan <[email protected]>

Fix lint

16098bf

Signed-off-by: lubronzhan <[email protected]>

lubronzhan force-pushed the topic/lubron/fix-61 branch from a733ff0 to 16098bf Compare February 4, 2024 03:16

lubronzhan changed the title ~~Support reconciling loadbalancerClass~~ Support loadbalancerClass Feb 4, 2024

lubronzhan force-pushed the topic/lubron/fix-61 branch from c71b570 to 953caf2 Compare February 4, 2024 06:39

More finalizer logic

47548a7

Signed-off-by: lubronzhan <[email protected]>

lubronzhan force-pushed the topic/lubron/fix-61 branch from 953caf2 to 47548a7 Compare February 4, 2024 07:10

lubronzhan requested a review from thebsdbox February 4, 2024 22:43

lubronzhan force-pushed the topic/lubron/fix-61 branch from 32e603d to e9a6dc8 Compare February 5, 2024 04:39

Add more check on the service type LB update event

a41b3a1

Signed-off-by: lubronzhan <[email protected]>

lubronzhan force-pushed the topic/lubron/fix-61 branch from e9a6dc8 to a41b3a1 Compare February 5, 2024 05:56

lubronzhan merged commit 9ef8a7a into kube-vip:main Feb 9, 2024
5 checks passed

lubronzhan deleted the topic/lubron/fix-61 branch February 9, 2024 06:23

lubronzhan mentioned this pull request Feb 9, 2024

how to allocate ip address when multiple lb networks in one namespace? #67

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support loadbalancerClass #100

Support loadbalancerClass #100

lubronzhan commented Jan 31, 2024 •

edited

Loading

thebsdbox commented Feb 6, 2024

lubronzhan commented Feb 6, 2024

Support loadbalancerClass #100

Support loadbalancerClass #100

Conversation

lubronzhan commented Jan 31, 2024 • edited Loading

Summary

Manual Test

Without loadbalancerclass

With loadbalancerclass

thebsdbox commented Feb 6, 2024

lubronzhan commented Feb 6, 2024

lubronzhan commented Jan 31, 2024 •

edited

Loading