Received message larger than max: PubSub error and message loop

[rdsubhas]

Reproduce Steps:

• Publish message > 4MB to a topic
• Retrieve message through subscription: Error: Received message larger than max (xxxxx vs. 4194304)

References

• pubsub: gRPC max recv set to 4MB google-cloud-node#2190
• Fix max message size error mia-0032/fluent-plugin-gcloud-pubsub-custom#37

Impact

I believe gRPC introduced a request limit of 4MB. But PubSub quotas clearly says, maximum message size can be 10MB. So many people already utilize this quota as a guidance, and publish messages upto 10MB. Example.

Because the quota is broken out-of-the-box by PubSub ruby client (which uses gRPC internally in one of the latest releases I think), messages don't get acked, and circulate in the system for 1 week (PubSub has fixed 1 week retention policy). So put all together, the backpressure is really bad. We had to drop all our subscriptions twice now in the course of debugging this error.

[blowmage]

Thanks for the issue. We'll investigate this and let you know what we find.

[blowmage]

@murgatroid99 It looks like a possible solution is to enable the following settings on the GRPC client:

'grpc.max_send_message_length': -1
'grpc.max_receive_message_length': -1

Is that reasonable? If so, are these names and values supposed to be provided in the channel_args argument?

[murgatroid99]

Yes, that should do what you want, and they are supposed to be provided in the channel_args argument (specifically, the third argument to the Client constructor). I'm surprised you're having problems with this. I thought this was fixed in #2192.

[blowmage]

Well, it was fixed for node, but not for Ruby. :)

Can we set the channel args on GRPC::Core::Channel? I'm having a hard time finding docs for it. The GAPIC layer takes care of creating a GRPC client for us, but we provide a channel object as an argument.

[murgatroid99]

That's what I get for not paying attention to which repo I'm looking at. Yes, when constructing a client stub, those should be passed in a hash in the channel_args keyword argument. When constructing a Channel directly, it looks like that goes into the second argument.

[blowmage]

Thank you for the confirmation. I have reproduced this error, added the channel_args configuration, and verified the fix. Opening a PR for this change.

[ravindranathakila]

This happens in the Python SDK as well. Is this an SDK error?

[timanovsky]

I think this also applies to Datastore, when using find_all result can be large. Although it is not documented well, what is the server->client maximal packet size, I'd assume it is 10MB similar to client->server messages.