Windows-specific read_registry / _win32_is_nic_enabled error on AWS EC2

[Jeff17Robbins]

Specifically, the experimentally derived test for a 1 on internal Registry value ConfigFlags doesn't work. The code erroneously thinks that a disabled network adapter is enabled. Based on this error, it uses the wrong DNS server address.

dnspython/dns/resolver.py

Line 938 in 585add9

def _win32_is_nic_enabled(self, lm, guid, _):

Microsoft documents that ConfigFlags is for internal use only.

Additionally, Microsoft has a supported API GetNetworkParams, that has been available since Windows 2000, to get the DNS nameservers.

I'd like to suggest that we consider using GetNetworkParams instead of the Windows Registry code to produce the list of nameservers.

I've attached a zipped standalone Python script that uses GetNetworkParams to produce the list of DNS nameservers on a Windows machine. I've tested it on a Windows 10 desktop and an AWS EC2 running Windows Server 2019 Datacenter Version 1809. It relies on the built-in ctypes package to access GetNetworkParams. The attached code isn't a pull request, as I wanted to gauge the project's level of interest in fixing this bug first.

win_dns_nameservers.zip

[rthalley]

We are definitely open to PRs to do things better on Windows.

Re the attached zip file, I'd prefer something written from scratch that we could ship under the dnspython ISC license rather than adapting code under another license.

[Jeff17Robbins]

Great! I'd like to participate. I've read LICENSE but the two sections quickly got me confused. If I write code from scratch, submit the PR, and it gets accepted, who owns the copyright? There are three different copyright assertions:

1. Copyright (C) Dnspython Contributors
2. Copyright (C) 2001-2017 Nominum, Inc.
3. Copyright (C) Google Inc.

[edit]
If my code is useful, I'd be happy to be part of "Dnspython Contributors", but am not interested in donating my code to Google.

@rthalley could you please provide more clarity about "the dnspython ISC license" you referred to? I would like to comply and get this code fixed. Thanks!

[rthalley]

Your code would not be donated to Google. The LICENSE file text about Nominum and Google is just mentioning the other copyrights, basically MIT-style, of some code contributed by Nominum and Google before we started using the ISC license.

See

https://opensource.org/licenses/ISC

for info on the ISC license.

If you don't put a copyright and license on the file, I will apply the "Dnspython Contributors" text to it.

You can put a copyright and ISC license with your name on the code you are contributing if you'd like. I'd then add the "Dnspython Contributors" copyright to it to cover any subsequent changes we made to it, but your copyright and license would remain in the file.

[rthalley]

See the #722 PR for my plans in this area. Also note my theory about why the ConfigFlags test failed. If this is something you can check in your AWS environment, I'd be interested to know.

[rthalley]

This PR has been merged.

[rthalley]

Note you need to "pip install WMI" to get the more accurate way

[jrobbins-LiveData]

I'll try to test it out this weekend.

[jrobbins-LiveData]

@rthalley once I ensured _have_wmi==True in win32util.py by copying pythoncom39.dll and pywintypes39.dll into my virtual environment's Scripts folder, I got the correct DNS server IP address on an AWS EC2 running Windows.

This is my test script

import dns.resolver

dns_resolver = dns.resolver.Resolver()
print(dns_resolver.nameservers)

Should I have expected pip install git+https://github.com/rthalley/dnspython.git to have done the dll file copy for me? Or is there a subsequent install step I needed to do, in lieu of manually copying those two dlls from .venv\Lib\site-packages\pywin32_system32?

[rthalley]

"pip install wmi" is the way to install what dnspython needs. Installing dnspython with pip does not currently install WMI as we're still testing if it works well enough to be the default.

If you are feeling adventurous, I am curious what happens if you edit win32util.py and make the following changes:

On line 7, set _prefer_wmi = False

On line 188, change "flags & 0x1" to "flags & 0x03".

Then rerun your test and see if it did the right thing. I think the WMI way is the best way, but I'm curious about which part of the ConfigFlags technique was wrong :). No worries if you do not wish to do this.

[rthalley]

Oh, and thanks for doing the test! It's super helpful!

[jrobbins-LiveData]

Unfortunately, ConfigFlags is 0 in all three "interfaces", so it doesn't matter whether the mask is 0x01 or 0x03.

Debugging the code from get_dns_info()

def get_dns_info():
	"""Extract resolver configuration."""
	getter = _getter_class()
	return getter.get()

getter.get() ends up in this loop, looping through the interfaces (I added a print)

while True:
	try:
		guid = winreg.EnumKey(interfaces, i)
                print(f"{guid=}")
		i += 1
		key = winreg.OpenKey(interfaces, guid)
		try:
			if not self._is_nic_enabled(lm, guid):
				continue
			self._config_fromkey(key, False)

and then, in _is_nic_enabled(), I added another print

key = r'SYSTEM\CurrentControlSet\Enum\%s' % pnp_id
print(f"{key=} {flags=}")
return not flags & 0x3

You can see that flags is 0 in each case. The first guid is the one that returns the seemingly erroneous '172.31.0.2'

guid='{1248d185-7737-4822-b94e-eeafd9a0d615}'
key='SYSTEM\CurrentControlSet\Enum\PCI\VEN_1D0F&DEV_EC20&SUBSYS_00000000&REV_00\3&13c0b0c5&1&28' flags=0
guid='{54b31d7e-36bf-4bbe-9ab2-106a939cd78c}'
key='SYSTEM\CurrentControlSet\Enum\XENVIF\VEN_XS0001&DEV_NET&REV_0000000B\0' flags=0
guid='{82d00445-1aae-424b-98e1-9f57b24a6942}'
key='SYSTEM\CurrentControlSet\Enum\ROOT\KDNIC\0000' flags=0
guid='{c0100896-e869-11e8-81bf-806e6f6e6963}'
['172.31.0.2', '10.0.0.2']

I don't know enough about the AWS EC2 environment to understand which of these interfaces is the one we ought to be looking at.

However, when I run ipconfig /all, all I see is

DNS Servers . . . . . . . . . . . : 10.0.0.2

Likewise, when running this in Powershell

Get-WmiObject -Class Win32_NetworkAdapterConfiguration | Select DnsServerSearchOrder

DnsServerSearchOrder

{10.0.0.2}

Leading me to think that the first interface (in the Enum\\PCI registry namespace) is to be avoided on a virtual machine. Which, speculatively, might make sense, since a VM (like an AWS EC2) isn't using the actual PCI bus NIC, but, rather, is going through some virtual interface.

[jrobbins-LiveData]

Regarding the pip install WMI problem I had on the AWS EC2

The problem is caused by the EC2 coming "pre-equipped" with a version of Python (or some of its DLLs) in a folder that is in the Path.

This System path setting apparently preempts the technique pywin32 is using to find its DLLs.

I think pywin32.pth plays a role

# .pth file for the PyWin32 extensions
win32
win32\lib
Pythonwin
# And some hackery to deal with environments where the post_install script
# isn't run.
import pywin32_bootstrap

Copying pythoncom39.dll and pywintypes39.dll in the the .venv\Scripts folder fixes the issue. As does removing the Amazon entry from the System Path. But I'm thinking pywin32 might be buggy in this case. I'll report that there, since I do not think this is a dnspython issue.

[rthalley]

Thanks for the experiment. It seems that the issue is that the old code is just finding everything instead of finding the same DNS settings that windows picks.

[jrobbins-LiveData]

Agreed.

I just finished a much simpler approach to solving this problem (obviating the registry and WMI). I'd be happy to donate it to the dnspython project.

It uses ctypes, which comes with Python, and uses DnsQueryConfig, a Windows API that is much simpler to call than GetNetworkParams.

(I also did a "clean room" version of calling GetNetworkParams using cffi, by way of comparison. But the DnsQueryConfig API seems to be a much better choice.)

Please give this file a look and tell me what you think?

[rthalley]

We need the domain and search list too. It looks like the DnsQueryConfig technique can get the domain, but from the documentation it looks like DnsConfigSearchList is not implemented. I remember looking at this call. I rejected GetNetworkParams as it looked very complicated to call. The WMI method, while probably absurdly complicated internally, was really easy to call and gave me high-level output. If I've missed something and we can get the search list, let me know.

[jrobbins-LiveData]

The other file in my prototype repo uses GetNetworkParams. Yes, it is more complicated, but not, in my opinion, hideously so. And since I got bit by that pywin32 problem on my EC2, I'm probably more nervous about the pythoncom WMI approach than a "straightforward" C API. Anyway, my working example of GetNetworkParams is here.

Unfortunately, I'm a DNS dunce, and don't know what the search list is! Do you have any pointer(s) so I can read up on that?
[edit] Is that list what interface.DNSDomainSuffixSearchOrder returns from WMI?

[another edit]
Also, does the code below show accessing the three things ("domain", "nameservers", and "search list") that you referred to? If so, I think you are right. I can't retrieve the domain name from GetNetworkParams... it seems to be returning an empty string. And, searching the web, it seems that only the registry (or WMI) know the "search list".

So, I think I should focus on getting the pywin32 virtual environment issue fixed. Without some change, the WMI solution for dnspython will fail if used in a virtualenv on an AWS EC2.

import dns.resolver

dns_resolver = dns.resolver.Resolver()

print(f"{dns_resolver.domain=}")
print(f"{dns_resolver.nameservers=}")
print(f"{dns_resolver.search=}")

[rthalley]

Yes, the sample code shows the three things we're trying to get. And yes, the search list I mentioned is what Windows calls the DnsSuffixSearchOrder. I don't see it in GetNetworkParams either.

Historically, the search list was used to save typing. If you typed in a name (typically one without dots though I guess XP might have done the wrong thing), then the local DNS resolver would try to lookup names with the various suffixes on the search list, and take the first one that existed. E.g. if you said "ping the_server" and the search list was a.example.com, b.example.com, then it would lookup the_server.a.example.com, and if it didn't exist, would look up the_server.b.example.com. The actual rules are a bit more complicated, see _get_qnames_to_try() in resolver.py if curious.

See e.g. DnsSuffixSearchOrder

It's really better NOT to use the search list, and dnspython doesn't use it by default now with its resolve() method, but some use cases need it so we support it. Also the existing registry scraping code supports it, so I wouldn't want to give it up.

Getting pywin32 fixed seems like a good plan.

[jrobbins-LiveData]

I opened an issue mhammond/pywin32#1803. They might view it as an AWS issue (which would be fair), but it shows potential pitfalls for consumers of pywin32, like WMI and, therefore, dnspython.