Getting occasional dns lookup timeouts when load testing, which are not being retried

[williamcoates]

What would you like to discuss?

I am load testing our platform, and in a run I usually get a handful of dns lookup errors (since it opens socket but doesnt finish the lookup):

{
  "name":"TimeoutError",
  "code":"ETIMEDOUT",
  "timings":
     {
        "start":1612969156746,
        "socket":1612969156746,
        "error":1612969166750,
        "phases":{"wait":0,"total":10004}
    },
 "event":"request"
 }

I have this configuration:

    got.extend({
      dnsCache: cacheable,
      cache: httpCache,
      responseType: 'json',
      resolveBodyOnly: true,
      retry: {
        limit: 5,
        maxRetryAfter: 10000
      },
      timeout: {
        lookup: 1000,
    })

I was trying to use got to retry after one second if there is a lookup failure, but when I add retry hooks I don't see any retries being attempted Confirmed that some dns failures do retry successfully, but I still get some DNS failures that timeout after a couple of minutes, which do not trigger any retries (no retry hooks logged)

Checklist

• I have read the documentation.

[szmarczak]

It's hard to say what's going on without a reproducible code. You may want to have a look at https://github.com/sindresorhus/got#dnscache

[williamcoates]

Thanks for the fast response @szmarczak ! The dnsCache is just new CacheableLookup(fallbackDuration: 0). Unfortunately i still get dns failures which is odd. My guess is because it respects the TTL which is coming from the ELB of 60s (I cannot change that unfortunately).

I will investigate more...

[szmarczak]

What tool are you using to stress test your app? What about your DNS server - does it handle your load?

You can limit the retry time via a calculateDelay option.

[williamcoates]

We are using flood.io to stress test. The DNS server is AWS route 53, using private dns within the VPC, so together with the dns caching I would assume that wouldn't be the problem.

I configured just a lookup timeout now, and I can see that worked a couple of times, and the retry hook showed it retrying and then succeeding which is great.

But I still get three dns timeouts where the retry hook was not invoked a single time which is odd, and the timeout is about 2 minutes which is also peculiar.

[williamcoates]

So what seems to be happening is that the socket is opened, but for whatever reason the timeout is not setup for the dns lookup sometimes. Could it be related to

got/source/core/utils/timed-out.ts

Line 121 in d0cd709

if (socket.connecting) {

?

Seems unlikely as I assume the idea for this is if its already connected, no need for the timeout.

[szmarczak]

Good point! I'll check that ASAP.

[szmarczak]

By the way, what's your Node.js version?