pubsub: No errors exposed for resource exhausted errors

[jameshartig]

Client

PubSub (latest master: )

Describe Your Environment

CentOS 7 on GCE

Expected Behavior

I expect an error from Receive if there is a resource exhaustion on StreamingPull requests.

Actual Behavior

Receive doesn't return an error and I assume silently retries.

We apparently hit our concurrent StreamingPull limit (120) months ago and never knew because Receive never returned an error. When we spun up more servers today they were only receiving 1/10th the amount of messages the other servers were and we couldn't figure out why until we happened to notice the Pub/Sub API dashboard showed 99.96% of StreamingPull calls errored.

[jba]

@fastest963 Was the error code ResourceExhausted?

Assuming the answer is yes:
@kamalaboulhosn @kir-titievsky We do indeed retry on ResourceExhausted. Should we revisit that?

[jameshartig]

@jba I'm not sure but I think so because the attached screenshot below shows 429 as the highest error. When I raised the limit the number of open connections jumped and the number of 429 errors dropped.

The following screenshot is limited to StreamingPull method.

[jeanbza]

Friendly ping @kamalaboulhosn re: revisiting retrying ResourceExhausted

[kamalaboulhosn]

What probably makes sense here is to not treat ResourceExhausted as retryable for streaming pull. For other resource limits, they are generally rate limits and not count limits. The streaming pull open connections limit is the only one that is an actual max count. Retrying isn't likely to help here given that connections are long-lived.

What do you think?

[jeanbza]

SG, thanks! https://code-review.googlesource.com/c/gocloud/+/36050