Skip to content

Commit

Permalink
add event when staticGatewayConfiguration validation failed
Browse files Browse the repository at this point in the history
  • Loading branch information
jwtty committed Jun 27, 2024
1 parent 8c112e5 commit 9d16cb5
Show file tree
Hide file tree
Showing 13 changed files with 37 additions and 25 deletions.
1 change: 1 addition & 0 deletions .codespellignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
aks
AfterAll
2 changes: 1 addition & 1 deletion .github/workflows/codespell.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# https://github.com/codespell-project/actions-codespell
# https://github.com/codespell-project/codespell
name: codespell
on: [push, pull_request]
on: [pull_request]
permissions:
contents: read

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ If `provisionPublicIps` is false, `egressIpPrefix` will be a list of private IPs

### Deploy a Pod using Static Egress Gateway

Contructing a pod to use a static egress gateway is simple: just add pod annotation `kubernetes.azure.com/static-gateway-configuration: <StaticGatewayConfiguration name>`. Only name is required here because kube-egress-gateway CNI plugin always assume the gateway is in the same namespace as the pod. Note that existing pods must be recreated to enable egress gateway because CNI plugin can only take effect when pod is being created. See sample pod [here](docs/samples/sample_pod.yaml).
Constructing a pod to use a static egress gateway is simple: just add pod annotation `kubernetes.azure.com/static-gateway-configuration: <StaticGatewayConfiguration name>`. Only name is required here because kube-egress-gateway CNI plugin always assume the gateway is in the same namespace as the pod. Note that existing pods must be recreated to enable egress gateway because CNI plugin can only take effect when pod is being created. See sample pod [here](docs/samples/sample_pod.yaml).

## Troubleshooting

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@ metadata:
app.kubernetes.io/name: gatewaylbconfiguration
app.kubernetes.io/instance: gatewaylbconfiguration-sample
app.kubernetes.io/part-of: kube-egress-gateway
app.kuberentes.io/managed-by: kustomize
app.kubernetes.io/managed-by: kustomize
app.kubernetes.io/created-by: kube-egress-gateway
name: gatewaylbconfiguration-sample
spec:
# TODO(user): Add fields here
gatewayNodepoolName: gwnodepool
gatewayVmssProfile: {}
provisionPublicIps: true
10 changes: 8 additions & 2 deletions config/samples/egressgateway_v1alpha1_gatewaystatus.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,14 @@ metadata:
app.kubernetes.io/name: gatewaystatus
app.kubernetes.io/instance: gatewaystatus-sample
app.kubernetes.io/part-of: kube-egress-gateway
app.kuberentes.io/managed-by: kustomize
app.kubernetes.io/managed-by: kustomize
app.kubernetes.io/created-by: kube-egress-gateway
name: gatewaystatus-sample
spec:
# TODO(user): Add fields here
readyGatewayConfigurations:
- interfaceName: wg-6000
staticGatewayConfiguration: default/staticgatewayconfiguration-sample
readyPeerConfigurations:
- interfaceName: wg-6000
podEndpoint: default/podendpoint-sample
publicKey: **********
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ metadata:
labels:
app.kubernetes.io/name: gatewayvmconfiguration
app.kubernetes.io/instance: gatewayvmconfiguration-sample
app.kubernetes.io/part-of: kube-egress-gateway
app.kubernekubernetestes.io/part-of: kube-egress-gateway
app.kuberentes.io/managed-by: kustomize

Check failure on line 8 in config/samples/egressgateway_v1alpha1_gatewayvmconfiguration.yaml

View workflow job for this annotation

GitHub Actions / Check for spelling errors

kuberentes ==> kubernetes
app.kubernetes.io/created-by: kube-egress-gateway
name: gatewayvmconfiguration-sample
spec:
# TODO(user): Add fields here
gatewayNodepoolName: gwnodepool
gatewayVmssProfile: {}
provisionPublicIps: true
2 changes: 1 addition & 1 deletion controllers/manager/gatewaylbconfiguration_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -471,7 +471,7 @@ func (r *GatewayLBConfigurationReconciler) reconcileLBRule(
}

if !needLB && deleteFrontend {
log.Info(fmt.Sprintf("no more gateway profile refering vmss(%s), deleting frontend and backend", names.frontendName))
log.Info(fmt.Sprintf("no more gateway profile referring vmss(%s), deleting frontend and backend", names.frontendName))
frontends := lb.Properties.FrontendIPConfigurations
for i, frontendConfig := range frontends {
if strings.EqualFold(to.Val(frontendConfig.ID), to.Val(frontendID)) {
Expand Down
4 changes: 2 additions & 2 deletions controllers/manager/gatewayvmconfiguration_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,8 @@ func ifLabelMatch(obj client.Object, m map[string]string) bool {
return true
}

lables := obj.GetLabels()
if v, ok := lables[labelKey]; ok {
labels := obj.GetLabels()
if v, ok := labels[labelKey]; ok {
// Return early if no labelValue was set.
if labelValue == "" {
return true
Expand Down
16 changes: 8 additions & 8 deletions controllers/manager/gatewayvmconfiguration_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -629,7 +629,7 @@ var _ = Describe("GatewayVMConfiguration controller unit tests", func() {
mockVMSSVMClient.EXPECT().List(gomock.Any(), testRG, vmssName).Return(vms, nil)
mockVMSSVMClient.EXPECT().Update(gomock.Any(), testRG, vmssName, "0", gomock.Any()).
DoAndReturn(func(ctx context.Context, rg, vmssName, instanceID string, vm compute.VirtualMachineScaleSetVM) (*compute.VirtualMachineScaleSetVM, error) {
// during update, we don't fillin vm.OSProfile. Fill in here for test purpose
// during update, we don't fill in vm.OSProfile. Fill in here for test purpose
vm.Properties.OSProfile = &compute.OSProfile{
ComputerName: to.Ptr("test"),
}
Expand Down Expand Up @@ -678,7 +678,7 @@ var _ = Describe("GatewayVMConfiguration controller unit tests", func() {
mockVMSSVMClient.EXPECT().List(gomock.Any(), testRG, vmssName).Return(vms, nil)
mockVMSSVMClient.EXPECT().Update(gomock.Any(), testRG, vmssName, "0", gomock.Any()).
DoAndReturn(func(ctx context.Context, rg, vmssName, instanceID string, vm compute.VirtualMachineScaleSetVM) (*compute.VirtualMachineScaleSetVM, error) {
// during update, we don't fillin vm.OSProfile. Fill in here for test purpose
// during update, we don't fill in vm.OSProfile. Fill in here for test purpose
vm.Properties.OSProfile = &compute.OSProfile{
ComputerName: to.Ptr("test"),
}
Expand Down Expand Up @@ -710,7 +710,7 @@ var _ = Describe("GatewayVMConfiguration controller unit tests", func() {
mockVMSSVMClient.EXPECT().List(gomock.Any(), testRG, vmssName).Return([]*compute.VirtualMachineScaleSetVM{existingVM}, nil)
mockVMSSVMClient.EXPECT().Update(gomock.Any(), testRG, vmssName, "0", gomock.Any()).
DoAndReturn(func(ctx context.Context, rg, vmssName, instanceID string, vm compute.VirtualMachineScaleSetVM) (*compute.VirtualMachineScaleSetVM, error) {
// during update, we don't fillin vm.OSProfile. Fill in here for test purpose
// during update, we don't fill in vm.OSProfile. Fill in here for test purpose
vm.Properties.OSProfile = &compute.OSProfile{
ComputerName: to.Ptr("test"),
}
Expand All @@ -731,7 +731,7 @@ var _ = Describe("GatewayVMConfiguration controller unit tests", func() {
Expect(err).To(BeNil())
})

It("should configure vmss and vm without publicIPConfiguration and return vmss instace private IPs when ipPrefixID is empty", func() {
It("should configure vmss and vm without publicIPConfiguration and return vmss instance private IPs when ipPrefixID is empty", func() {
existingVMSS, expectedVMSS := getEmptyVMSS(), getConfiguredVMSSWithoutPublicIPConfig()
mockVMSSClient := az.VmssClient.(*mock_virtualmachinescalesetclient.MockInterface)
mockVMSSClient.EXPECT().CreateOrUpdate(gomock.Any(), testRG, vmssName, gomock.Any()).
Expand All @@ -746,7 +746,7 @@ var _ = Describe("GatewayVMConfiguration controller unit tests", func() {
mockVMSSVMClient.EXPECT().List(gomock.Any(), testRG, vmssName).Return(vms, nil)
mockVMSSVMClient.EXPECT().Update(gomock.Any(), testRG, vmssName, "0", gomock.Any()).
DoAndReturn(func(ctx context.Context, rg, vmssName, instanceID string, vm compute.VirtualMachineScaleSetVM) (*compute.VirtualMachineScaleSetVM, error) {
// during update, we don't fillin vm.OSProfile. Fill in here for test purpose
// during update, we don't fill in vm.OSProfile. Fill in here for test purpose
vm.Properties.OSProfile = &compute.OSProfile{
ComputerName: to.Ptr("test"),
}
Expand All @@ -763,7 +763,7 @@ var _ = Describe("GatewayVMConfiguration controller unit tests", func() {
Expect(err).To(BeNil())
})

It("should remove vmss and vm publicIPConfiguration and return vmss instace private IPs when ipPrefixID is updated to be empty", func() {
It("should remove vmss and vm publicIPConfiguration and return vmss instance private IPs when ipPrefixID is updated to be empty", func() {
existingVMSS, expectedVMSS := getConfiguredVMSSWithNameAndUID(), getConfiguredVMSSWithoutPublicIPConfig()
mockVMSSClient := az.VmssClient.(*mock_virtualmachinescalesetclient.MockInterface)
mockVMSSClient.EXPECT().CreateOrUpdate(gomock.Any(), testRG, vmssName, gomock.Any()).
Expand All @@ -780,7 +780,7 @@ var _ = Describe("GatewayVMConfiguration controller unit tests", func() {
mockVMSSVMClient.EXPECT().List(gomock.Any(), testRG, vmssName).Return(vms, nil)
mockVMSSVMClient.EXPECT().Update(gomock.Any(), testRG, vmssName, "0", gomock.Any()).
DoAndReturn(func(ctx context.Context, rg, vmssName, instanceID string, vm compute.VirtualMachineScaleSetVM) (*compute.VirtualMachineScaleSetVM, error) {
// during update, we don't fillin vm.OSProfile. Fill in here for test purpose
// during update, we don't fill in vm.OSProfile. Fill in here for test purpose
vm.Properties.OSProfile = &compute.OSProfile{
ComputerName: to.Ptr("test"),
}
Expand Down Expand Up @@ -816,7 +816,7 @@ var _ = Describe("GatewayVMConfiguration controller unit tests", func() {
mockVMSSVMClient.EXPECT().List(gomock.Any(), testRG, vmssName).Return([]*compute.VirtualMachineScaleSetVM{existingVM}, nil)
mockVMSSVMClient.EXPECT().Update(gomock.Any(), testRG, vmssName, "0", gomock.Any()).
DoAndReturn(func(ctx context.Context, rg, vmssName, instanceID string, vm compute.VirtualMachineScaleSetVM) (*compute.VirtualMachineScaleSetVM, error) {
// during update, we don't fillin vm.OSProfile. Fill in here for test purpose
// during update, we don't fill in vm.OSProfile. Fill in here for test purpose
vm.Properties.OSProfile = &compute.OSProfile{
ComputerName: to.Ptr("test"),
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ func (r *StaticGatewayConfigurationReconciler) reconcile(
defer func() { mc.ObserveControllerReconcileMetrics(succeeded) }()

if err := validate(gwConfig); err != nil {
r.Recorder.Event(gwConfig, corev1.EventTypeWarning, "InvalidSpec", err.Error())
return err
}

Expand Down
6 changes: 3 additions & 3 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This document provides steps to troubleshoot pod outbound connectivity issue when kube-egress-gateway is in use. Troubleshooting contains two steps:
1. Validate static egress gateway is successfully provisioned and egress traffic has right source IP.
2. Validate pod provisoning and pod-gateway connectivity.
2. Validate pod provisioning and pod-gateway connectivity.

## StaticGatewayConfiguration Validation

Expand Down Expand Up @@ -100,7 +100,7 @@ After checking the CR objects, you can login to the gateway node and check netwo
10.243.0.5 dev host0 scope link
10.244.0.14 dev wg-6000 scope link # expect to see pod's IP routed via the wg-* interface.
```
For iptables-rules, there are several rules added to masquarade packets. The target IP is the private IP of the `StaticGatewayConfiguration` private IP.
For iptables-rules, there are several rules added to masquerade packets. The target IP is the private IP of the `StaticGatewayConfiguration` private IP.
The rule names should have suffix of the `status.gatewayServerProfile.port`, same as the wireguard interface.
```bash
$ ip netns exec ns-static-egress-gateway iptables-save # traffic is masqueraded
Expand Down Expand Up @@ -131,7 +131,7 @@ After checking the CR objects, you can login to the gateway node and check netwo
latest handshake: 10 minutes, 38 seconds ago
transfer: 11.43 KiB received, 11.57 KiB sent
```
*Note: if `wg` command does not exist, run `apt isntall wireguard-tools` to install*
*Note: if `wg` command does not exist, run `apt install wireguard-tools` to install*

## Pod Validation

Expand Down
4 changes: 2 additions & 2 deletions helm/kube-egress-gateway/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Azure cloud configuration provides information including resource metadata and c
| `config.azureCloudConfig.userAgent` | The userAgent provided to Azure when accessing Azure resources. | |
| `config.azureCloudConfig.location` | The azure region where resource group and its resources is deployed. | |
| `config.azureCloudConfig.gatewayLoadBalancerName` | The name of the load balancer in front of gateway VMSS for high availability. | Required, helm chart defaults to `kubeegressgateway-ilb`. |
| `config.azureCloudConfig.loadBalancerResourceGroup` | The resouce group where the load balancer to be deployed. | Optional. If not provided, it's the same as `config.azureCloudConfig.resourceGroup`. |
| `config.azureCloudConfig.loadBalancerResourceGroup` | The resource group where the load balancer to be deployed. | Optional. If not provided, it's the same as `config.azureCloudConfig.resourceGroup`. |
| `config.azureCloudConfig.vnetName` | The name of the virtual network where load balancer frontend ip comes from. | |
| `config.azureCloudConfig.vnetResourceGroup` | The resource group where the virtual network is deployed. | Optional. If not set, it's the same as `config.azureCloudConfig.resourceGroup`. |
| `config.azureCloudConfig.subnetName` | The name of the subnet inside the virtual network where the load balancer frontend ip comes from. | |
Expand Down Expand Up @@ -75,7 +75,7 @@ config:
## common configurations
The Helm chart installs 5 components with different images: gateway-controller-manager, gateway-daemon-manager, gateway-CNI-manager, gateway-CNI, and gateway-CNI-Ipam. For easy deployment, we provide `common.imageRepository` and `common.imageTag`. If all images can be downloaded from the same reposity (use `common.imageRepository`) and come from the same build/release (use `common.imageTag`), you can set just these two values instead of each individual value as shown in the following sections. If for any particular components, you want to apply a different repository/tag, you can specify in their own configurations, overriding these two values.
The Helm chart installs 5 components with different images: gateway-controller-manager, gateway-daemon-manager, gateway-CNI-manager, gateway-CNI, and gateway-CNI-Ipam. For easy deployment, we provide `common.imageRepository` and `common.imageTag`. If all images can be downloaded from the same repository (use `common.imageRepository`) and come from the same build/release (use `common.imageTag`), you can set just these two values instead of each individual value as shown in the following sections. If for any particular components, you want to apply a different repository/tag, you can specify in their own configurations, overriding these two values.

Additionally, `common.gatewayLbProbePort` defines the gateway LoadBalancer probe port which is consumed by both gateway-controller-manager (LB probe creator) and gateway-daemon-manager (probe server). The default value is `8082`.

Expand Down
2 changes: 1 addition & 1 deletion pkg/cni/ipam/delegate.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func (wrapper *IPWrapper) WithIP(configFunc func(ipamResult *current.Result) err
if err != nil {
recoverErr := wrapper.DeleteIP()
if recoverErr != nil {
err = fmt.Errorf("error occured %w and failed to delete ip: %s", err, recoverErr.Error())
err = fmt.Errorf("error occurred %w and failed to delete ip: %s", err, recoverErr.Error())
}
}
}()
Expand Down

0 comments on commit 9d16cb5

Please sign in to comment.