Event loop blocked

[handou-o]

Hello, I got some weird issues when I try to publish a lot of messages to topics with a promise.all

Here is part of my code:

 async getOrCreateTopic(topicName: string) : Promise<Topic> {
    let topic = this.client.topic(topicName);
    const [exists] = await topic.exists();
    if (!exists) {
      const created = await topic.create();
      topic = created[0];
    }
    return topic;
  }
  
async publish(
    topicName: string,
    message: MessageType,
    attributes: Record<string, string> = {},
  ): Promise<void> {
        const topic = await this.getOrCreateTopic(topicName);
      await topic.publishMessage({
        json: message,
        attributes,
      });
}

test() {
    const promises: Promise<void>[] = [];
    for (let i = 0; i < 10; i++) {
      promises.push(publisher.publish('test.pubsubTopic', { foo: 'foo' }));
    }
    await Promise.all(promises);
}

When I execute the test function sometimes randomly the process got stuck and never finish, it seems to happen more often when the topic does not exist, It happens on emulator and on a real GCP project.

[feywind]

@handou-o Is your test() also async? How is that being called? I'm asking because, with async and await, it's Promises all the way down, so sometimes things can get hung up near the bottom of that stack. (e.g. something unrelated might be blocking your event loop, and these Promises don't get to complete.)

[handou-o]

yes, sorry I forgot to put it on my example but yeah obviously test is async as well.
I'm quite sure that it's the method test that block the event loop as I ran that code without anything else

[scagood]

Are you using node 18? I ask because we started to notice similar issues when moving from node 16 to node 18.

Our issues are around subscriptions though, not around topics/publishing. We know that its not just a promise not resolving because we also have an interval setup to log processed messages, this also seems to just stop until our pods reboot themselves.

The interval is as simple as this:

const interval = setInterval(() => {
    console.info(`${processedMessageCount} messages processed.`);
    processedMessageCount = 0;
}, 300000);

[philwhln]

We're consistently seeing an issue with publishMessage. When we have high throughput and the number of concurrent messages waiting for pubsub publishing approaches 20, everything stops. The last call to publishMessage never returns and the CPU is maxed out.

[philwhln]

We're consistently seeing an issue with publishMessage. When we have high throughput and the number of concurrent messages waiting for pubsub publishing approaches 20, everything stops. The last call to publishMessage never returns and the CPU is maxed out.

This is with Node 18. We upgraded from Node 16 due to similar issues. We're now back on Node 14 for this application and so far (touch wood), it looks like this issue has gone away.

If someone can point me to the underlying code for publishMessage, I'd be happy to take a look and see if I can understand what the issue might be.

[handou-o]

@scagood yes it is on node 18,
The only solution that I found is to call getOrCreate before the loop and use the returned topic in the loop

[meriamBenSassi]

We had a similar issue using both node 16.19.0 and 18.13.0.
It seems to be an issue with node and http
nodejs/node#46234

[scagood]

This seems to have been resolved for us in node 18.12, and 18.15.

So I think this is probably not an issue with pubsub directly, so I think we can close this.

[philwhln]

This seems to have been resolved for us in node 18.12, and 18.15.

Great! We're also looking at moving back to 18, so this is good to hear. Was watching the Node releases and saw the fix come through.

So I think this is probably not an issue with pubsub directly, so I think we can close this.

Agreed. The breaking change in Node has been reverted.
nodejs/node#46234

[feywind]

Closing this since it sounds like it was an issue solved outside the library.