Helm - Headless Service, and DNS entry - POD cannot start

[Hoobciu]

in some cases, APIM pods cannot start.
displaying error:
[ErrorResolvingHostname], exception: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, ValueStopwatch stopwatch)
at System.Net.Dns.<>c.b__33_0(Object s, ValueStopwatch stopwatch)
at System.Net.Dns.<>c__DisplayClass39_01.<RunAsync>b__0(Task <p0>, Object <p1>) at System.Threading.Tasks.ContinuationResultTaskFromTask1.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.WindowsAzure.ApiManagement.Proxy.Runtime.DefaultDnsNameResolver.ResolveAsync(String name, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline.IO\Dns\DefaultDnsNameResolver.cs:line 26
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext), source: Neighborhood

Solution which solved the issue was to add parameter to service

azure-apim-gateway-azure-api-management-gate-instance-discovery

under .spec.

publishNotReadyAddresses: true

tells Kubernetes to register pod IP addresses in the service's DNS record EVEN IF the pods are not ready/healthy.
Without this setting, we had a circular dependency:

Pod needed DNS resolution to start
DNS resolution needed endpoints in the service
Service needed healthy pods
Pods couldn't be healthy because they couldn't start due to DNS...

With publishNotReadyAddresses: true:

Service publishes pod IPs immediately after pod creation
DNS resolution works before pod is ready
Pod can start successfully
The cycle is broken

This is a common pattern for bootstrap/discovery services where components need to find each other during startup.

[Nikkelmann]

Please note, this was created base on an active support ticket.

[tomkerkhove]

If there is a support ticket, then let's handle it through this channel. With the context above, I don't have enough information. For example, publishNotReadyAddresses should not be required but it also does not mention the SHGW version being used

[Nikkelmann]

We are using version 1.11.0 of the helm chart (https://artifacthub.io/packages/helm/azure-api-management/azure-api-management-gateway/1.11.0).

[Nikkelmann]

It had been working without any issues for the past year, then it crashed in two environments (production and test) this week and was not able to start up again without this workaround applied to the service.

We are still investigating the cause with MS support.

[tomkerkhove]

If there is active investigation, then let's close it here

[colincmac]

@tomkerkhove Was this resolved? I have a customer with the same issue and the publishNotReadyAddresses change fixed it for them as well. No support case yet, but if it requires a change in the Helm template, I can submit a PR.

• AKS is internal with egress outbound type LoadBalancer. No HTTP Proxy enabled.
• In Azure Gov
• Using the most recent Helm template & app version (greenfield)

I'm gathering more details and the logs, but we solved the issue by manually setting that value in the service to true during a debug session.

[tomkerkhove]

I honestly never got additional information nor reproduction steps so no idea =)

Adding this flag feels a bit like a hack to me and normally gateway should just continue so I'm hoping to verify the root cause before applying a mitigation/hack to unblock. Feel free to share repro steps when you have them

[colincmac]

I'll try to create a repo and share.