Customized Dialer passed to ClientConn got overwritten for unix domain socket

[hanlins]

Please see the FAQ in our main README.md, then answer the questions below before
submitting your issue.

What version of gRPC are you using?

First found issue in v1.27.0. Also tried v1.33.1 and latest commit a223251d8bced825764bf1e450b9840fd9227195, the issue still exists.

What version of Go are you using (go version)?

go1.14

What operating system (Linux, Windows, …) and version?

capv@test-4-md-0-74fb968bd8-zhjpq [ ~ ]$ uname -a
Linux test-4-md-0-74fb968bd8-zhjpq 4.19.145-2.ph3 #1-photon SMP Wed Sep 23 02:26:04 UTC 2020 x86_64 GNU/Linux

What did you do?

More context in kubernetes/kubernetes#95727 (comment).
During kubelet startup, it created a ClientConn object with a customized dialer (see here).

Added logging in gRPC dial to print out the address of the dial function being called, I found that before I restart containerd (which is the gRPC server that is accessed via a unix domain socket), the original customized dialer function was invoked. However after containerd restarted (the unix domain socket was disconnected for a second and then back online), the customized dialer was overwritten as a different value.

What did you expect to see?

Expect to the gRPC client always uses the passed customized dialer for transport.

What did you see instead?

The customized dialer passed to gRPC client was overwritten (maybe as of connection state reset).

[dfawley]

Nothing ever assigns to the dialer besides the With[Context]Dialer dial options, so I'm not sure how this could be possible. Are you sure this is the same ClientConn? Can you debug this a bit further or provide a way we can reproduce this ourselves?

Why are you using a custom dialer? After #3890, gRPC-Go now supports the unix scheme without the need for a custom dialer.

[stale]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.