Document the new Calico networking backend choice

docs/networking/calico.md

@@ -30,17 +30,46 @@ kops create cluster \
 
 ## Configuring
 
-### Enable Cross-Subnet mode in Calico (AWS only)
+### Select a Networking Backend
 
-Calico supports a new option for IP-in-IP mode where traffic is only encapsulated
-when it’s destined to subnets with intermediate infrastructure lacking Calico route awareness
-– for example, across heterogeneous public clouds or on AWS where traffic is crossing availability zones/ regions.
+In order to route network traffic to and from Kubernetes pods, Calico can use either of two networking backends: BIRD
+(for BGP) or VXLAN. With the BIRD backend, Calico runs the BIRD BGP daemon in each "calico-node" container to distribute
+routes to each machine. In this mode, Calico can use either IP-in-IP or VXLAN encapsulation between machines. IP-in-IP
+encapsulation requires the routes maintained with BGP, whereas VXLAN encapsulation does not.
 
-With this mode, IP-in-IP encapsulation is only [performed selectively](https://docs.projectcalico.org/v3.10/networking/vxlan-ipip#configure-ip-in-ip-encapsulation-for-only-cross-subnet-traffic).
+With the VXLAN backend, Calico does not run the BIRD daemon and does not use BGP to maintain routes. This rules out use
+of IP-in-IP encapsulation, and allows only VXLAN encapsulation.
+
+kops chooses the BIRD networking backend by default, it being the most flexible, which is equivalent to writing the following in the cluster spec:
+```yaml
+  networking:
+    calico:
+      backend: bird
+```
+To use the VXLAN networking backend instead, add the following to the cluster spec:
+```yaml
+  networking:
+    calico:
+      backend: vxlan
+```
+Though IP-in-IP encapsulation uses fewer bytes of overhead per packet than VXLAN encapsulation, [VXLAN can be a better
+choice when used in concert with Calico's eBPF
+dataplane|https://docs.projectcalico.org/maintenance/troubleshoot/troubleshoot-ebpf#poor-performance]. In particular,
+eBPF programs can redirect packets between Layer 2 devices, but not between devices at Layer 2 and Layer 3, as is required to use IP-in-IP
+tunneling.
+
+### Enable Cross-Subnet mode in Calico
+
+Calico supports a new option for both of its IP-in-IP and VXLAN encapsulation modes where traffic is only encapsulated
+when it’s destined to subnets with intermediate infrastructure lacking Calico route awareness—for example, across
+heterogeneous public clouds or on AWS where traffic is crossing availability zones.
+
+With this mode, encapsulation is only [performed selectively](https://docs.projectcalico.org/v3.10/networking/vxlan-ipip#configure-ip-in-ip-encapsulation-for-only-cross-subnet-traffic).
 This provides better performance in AWS multi-AZ deployments, and in general when deploying on networks where
 pools of nodes with L2 connectivity are connected via a router.
 
-Note that Calico by default, routes between nodes within a subnet are distributed using a full node-to-node BGP mesh.
+Note that by default with Calico—when using its BIRD networking backend—routes between nodes within a subnet are
+distributed using a full node-to-node BGP mesh.
 Each node automatically sets up a BGP peering with every other node within the same L2 network.
 This full node-to-node mesh per L2 network has its scaling challenges for larger scale deployments.
 BGP route reflectors can be used as a replacement to a full mesh, and is useful for scaling up a cluster. [BGP route reflectors are recommended once the number of nodes goes above ~50-100.](https://docs.projectcalico.org/networking/bgp#topologies-for-public-cloud)
@@ -55,15 +84,28 @@ To enable this mode in a cluster, add the following to the cluster spec:
     calico:
       crossSubnet: true
 ```
-In the case of AWS, EC2 instances have source/destination checks enabled by default.
-When you enable cross-subnet mode in kOps 1.19+, it is equivalent to:
+In the case of AWS, EC2 instances' ENIs have source/destination checks enabled by default.
+When you enable cross-subnet mode in kOps 1.19+, it is equivalent to either:
 ```yaml
   networking:
     calico:
       awsSrcDstCheck: Disable
+      backend: bird
       IPIPMode: CrossSubnet
+      VXLANMode: Never
 ```
-An IAM policy will be added to all nodes to allow Calico to execute `ec2:DescribeInstances` and `ec2:ModifyNetworkInterfaceAttribute`, as required when [awsSrcDstCheck](https://docs.projectcalico.org/reference/resources/felixconfig#spec) is set.
+or
+```yaml
+  networking:
+    calico:
+      awsSrcDstCheck: Disable
+      backend: vxlan
+      IPIPMode: Never
+      VXLANMode: CrossSubnet
+```
+depending on which networking backend you have selected.
+
+In AWS an IAM policy will be added to all nodes to allow Calico to execute `ec2:DescribeInstances` and `ec2:ModifyNetworkInterfaceAttribute`, as required when [awsSrcDstCheck](https://docs.projectcalico.org/reference/resources/felixconfig#spec) is set.
 For older versions of kOps, an addon controller ([k8s-ec2-srcdst](https://github.com/ottoyiu/k8s-ec2-srcdst))
 will be deployed as a Pod (which will be scheduled on one of the masters) to facilitate the disabling of said source/destination address checks.
 Only the control plane nodes have an IAM policy to allow k8s-ec2-srcdst to execute `ec2:ModifyInstanceAttribute`.