Set / document default envoy circuit_breakers

[devinrsmith]

Old title, TLDR: Leaking server-side StreamObservers leads to "overflow" failures

While testing out #1440 via a java client WIP, I noticed that I was not getting server-side responses to my input table requests. Turns out there is a server-side bug where the StreamObserver is not completing (to be fixed in #1565), essentially causing the server-side to "leak" StreamObservers.

For testing my java client, I continued on, and started ignoring the response from the server so I could continue on without the server-side fix.

Things seemed to work just fine, until I noticed that the first 1024 requests "worked" (I was able to see the input table changes in the UI), but future requests would not work. I was able to reproduce this 100% of the time by restarting the client.

I then instrumented my client to not ignore errors, and got back:

UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: overflow

after 1024 requests.

Either of the following "fixed" the issue

• Fixing the server side leak
• Bypassing envoy, and connecting directly to grpc-api server

When trying multiple clients at the same time, the overflow was reach when the sum-total of leaked observers reached 1024, which seemed to suggest that this was some sort of global envoy setting.

@niloc132 researched, and found https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/circuit_breaking:

Circuit breakers are enabled by default and have modest default values, e.g. 1024 connections per cluster. To disable circuit breakers, set the thresholds to the highest allowed values.

Given our push-based subscription model, this likely means that, by default, our envoy configuration would start failing once a server had 1024 subscriptions, and likely sooner due to other concurrent requests.

This patch:

index 0cc44fa2c..383e0ca53 100644
--- a/envoy/contents/envoy.yaml
+++ b/envoy/contents/envoy.yaml
@@ -88,6 +88,11 @@ static_resources:
                     socket_address:
                       address: grpc-api # here we assume the name of the websocket proxy
                       port_value: 8080
+      circuit_breakers:
+        thresholds:
+          max_connections: 1000000
+          max_pending_requests: 1000000
+          max_requests: 1000000
     - name: web
       connect_timeout: 10s
       type: LOGICAL_DNS

pushed that limit up to 1 million.

We'll likely want some documentation on the proper envoy circuit breaker limit depending on deployment needs.

[devinrsmith]

Proposal:

• turn the circuit-breaker "off" by default with a very high number (https://www.envoyproxy.io/docs/envoy/latest/faq/load_balancing/disable_circuit_breaking#faq-disable-circuit-breaking)
• document for production use cases it's probably a good idea to choose a number suitable for the use-case - to detect when things go wrong and/or guard against over-usage.