pubsub: add support for extending ACK deadline

[h27771420]

Is your feature request related to a problem? Please describe.

I guess I have to start with the differences between AWS SQS and GCP Pub/Sub, here are what I think are the more significant differences between them

1. SQS accepts Ack queue message after the visibility timeout is exceeded.
2. But it does not work for Pub/Sub. If the Ack Deadline is exceeded, our Ack operation will fail.
3. The upper limit of visibility timeout of SQS is 12 hours.
4. The upper limit of Ack deadline of Pub/Sub is 10 minutes.

So here is my current situation, I'm migrating our library from the AWS SDK to go-cloud to invoke Pub/Sub. (Eventually I would like to make it works like a multi-cloud/multi-queue-service support.) (Btw, I really have to say, go-cloud is truly amazing. 😄 )
But, the design logic in all my previous services was basically Queue-message-driven. I usually poll the message first and then start processing the message. When the processing is completed, I would decide whether to Ack or Nack.

I know that though they are essentially different services, the above differences are too huge, so when processing some time-consuming requests, 10 minutes is often not enough. (Pub/Sub also not allow us to Ack after reached the deadline. 😔 )
In the end my service usually can handle that message successfully, but because that message cannot be correctly Acked, that message will be judged as a processing failure by Pub/Sub, and will eventually be automatically transferred to the dead letter topic.

Describe the solution you'd like

So I hope go-cloud can add a new method to extend Ack deadline.

func (m *pubsub.Message) ExtendAckDeadline(d time.Duration) error {...}

Describe alternatives you've considered

The alternative I guess would be
For polling:

1. As soon as I poll the message, I'd need to cache it into Redis or somewhere, and like marking it is an on flying message.
   (Ah, a point just came into my mind. In that case, why don't I just stop using Pub/Sub in the first place? Eventually I still have to rewrite the entire message queue mechanism.)

For handling:

1. Pop out the message from that queue cache(or storage), handle it.
2. if succeeded, do nothing.
3. if failed, push it back into queue, but add 1 failure count.
4. if reached the max retry times, deliver it to the corresponding dead letter topic.

(However, there is still a problem. If it panic, the whole thing will be ruined, so another thing may be needed to provide to prevent that request from disappearing forever after panicing.)

Additional context

Sorry, I seem to have written context in the wrong place, please refer to above.

[vangent]

Ack; we've considered this before (#3507).

[h27771420]

@vangent Ahhhhh, thanks for your response, I guess you mean #3130? And that AckID should be abled to invoke ModifyAckDeadline.

I saw the reason, but in my case, it feels weird to open another google/pubsub client to make this request.
It is like opening the Pub/Sub client twice at the same time, I really don't like to do that. 😔

Or could it be an option in pubsub/gcppubsub to decide whether to automatically extend Ack deadline, or let the Subscription client extend a specific time for Ack deadline?
(e.g., A MaxExtension option in google/pubsub googleapis/google-cloud-go#608)

For go-cloud, maybe

  url := "gcppubsub://...?MaxExtension=3000" // sec

[vangent]

You can certainly do this with As.

• pubsub.Subscription.As() will get you a SubscriberClient (https://github.com/google/go-cloud/blob/master/pubsub/gcppubsub/gcppubsub.go#L45)
• pubsub.Message.As() will get you a pb.ReceivedMessage (https://github.com/google/go-cloud/blob/master/pubsub/gcppubsub/gcppubsub.go#L48) which contains the AckID in it.
• You can then use the SubscriberClient to call ModifyAckDeadline. It's not opening another client.

#3507 was more about modifying the gcppubsub provider to support automatically extending ack deadlines, similar to what the higher-level GCP pubsub client does (there are references in #3507). It doesn't look like this functionality makes sense for pubsub as a whole, but it might be reasonable for the GCP pubsub provider specifically. I will consider it; in the meantime if you're stuck you can do this yourself as described above.

[h27771420]

Awesome, thanks for pointing out the pubsub.Subscription.As(), I think this could be quite elegant.

• pubsub.Subscription.As() will get you a SubscriberClient (https://github.com/google/go-cloud/blob/master/pubsub/gcppubsub/gcppubsub.go#L45)

[vangent]

If you're happy with that solution, great. I'll add an example and call this done.

While it does seem possible to have the gcppubsub do this automatically for you, there doesn't seem to be much call for it, so I'm not sure it's worth the complexity, especially since I'd probably have to default it to "off" to avoid breaking any current users.