Cannot set "ExtraHosts" for edgeAgent with Windows Containers

[terrymandin]

Expected Behavior

ExtraHosts configured in createOptions should be available in the container.

Current Behavior

The name is not resolvable within the container

Steps to Reproduce

1. Update CreateOptions in the config.yaml for edgeAgent:

agent:
  name: "edgeAgent"
  type: "docker"
  env: {}
  config:
    image: "mcr.microsoft.com/azureiotedge-agent:1.1"
    auth: {}
    createOptions:
      hostConfig:
        extraHosts: ["tmbogusip.com:192.168.0.1"]

2. Update edgeAgent CreateOptions in the Runtime Settings when Setting Modules:

{
  "HostConfig": {
    "ExtraHosts": [
      "tmbogusip.com:192.168.0.1"
    ]
  }
}

3. Restart IoT Edge
4. Go into the container
   docker -H npipe:////./pipe/iotedge_moby_engine exec -it edgeAgent cmd
5. Ping the ipaddress

C:\app>ping tmbogusip.com
Ping request could not find host tmbogusip.com. Please check the name and try again.

6. Note that the name is not resolveable

Context (Environment)

Output of iotedge check

Click here

C:\Users\tmandin>iotedge check
Configuration checks
--------------------
√ config.yaml is well-formed - OK
√ config.yaml has well-formed connection string - OK
√ container engine is installed and functional - OK
√ Windows host version is supported - OK
√ config.yaml has correct hostname - OK
√ config.yaml has correct URIs for daemon mgmt endpoint - OK
√ latest security daemon - OK
√ host time is close to real time - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: certificates - Warning
    The Edge device is using self-signed automatically-generated development certificates.
    They will expire in 79 days (at 2021-07-22 23:38:05 UTC) causing module-to-module and downstream device communication to fail on an active deployment.
    After the certs have expired, restarting the IoT Edge daemon will trigger it to generate new development certs.
    Please consider using production certificates instead. See https://aka.ms/iotedge-prod-checklist-certs for best practices.
√ production readiness: container engine - OK
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its C:\Windows\Temp\edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its C:\Windows\Temp\edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with IoT Hub AMQP port - OK
√ host can connect to and perform TLS handshake with IoT Hub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with IoT Hub MQTT port - OK
√ container on the IoT Edge module network can connect to IoT Hub AMQP port - OK
√ container on the IoT Edge module network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to IoT Hub MQTT port - OK

16 check(s) succeeded.
5 check(s) raised warnings. Re-run with --verbose for more details.

Device Information

• Host OS: Windows 10 Enterprise N Version 1809 (OS Build 17763.1879)
• Architecture: amd64 (Windows Azure VM)
• Container OS: Windows containers

Runtime Versions

• aziot-edged: iotedge 1.1.1
• Edge Agent: 1.1
• Edge Hub: 1.1
• Docker/Moby: 19.03.12+azure

Logs

aziot-edged logs

N/A: In Event Viewer

edge-agent logs

<6> 2021-05-04 01:15:49.103 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<6> 2021-05-04 01:15:49.294 +00:00 [INF] - Installing certificates [CN=Test Edge Owner CA:7/22/2021 11:38:05 PM] to CertificateAuthority
<6> 2021-05-04 01:15:49.512 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: /metrics
<6> 2021-05-04 01:15:49.731 +00:00 [INF] - Updating performance metrics every 05m:00s
<6> 2021-05-04 01:15:49.736 +00:00 [INF] - Started operation Get system resources
<6> 2021-05-04 01:15:49.738 +00:00 [INF] - Collecting metadata metrics
<6> 2021-05-04 01:15:49.797 +00:00 [INF] - Set metadata metrics: 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"windows","Architecture":"x86_64","Version":"1.1.1 (27369573a0c04f9c2deeaf1ba4198a7efb8e4604)","Provisioning":{"Type":"manual.device_connection_string","DynamicReprovisioning":false},"ServerVersion":"19.03.12+azure","KernelVersion":"10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)","OperatingSystem":"Windows 10 Enterprise N Version 1809 (OS Build 17763.1879)","NumCpus":4,"Virtualized":"unknown"}, True
<6> 2021-05-04 01:15:49.819 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2021-05-04 01:15:49.826 +00:00 [INF] - Started operation refresh twin config
<6> 2021-05-04 01:15:49.851 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2021-05-04 01:15:50.289 +00:00 [INF] - Created persistent store at C:\Windows\TEMP\edgeAgent
<6> 2021-05-04 01:15:50.385 +00:00 [INF] - Started operation Metrics Scrape
<6> 2021-05-04 01:15:50.385 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2021-05-04 01:15:50.623 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2021-05-04 01:15:50.623 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2021-05-04 01:15:50.623 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2021-05-04 01:15:50.623 +00:00 [INF] - Registering request handler RestartModule
<6> 2021-05-04 01:15:51.042 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2021-05-04 01:15:51.256 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2021-05-04 01:15:51.371 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 9 and reported properties version 39.
<6> 2021-05-04 01:15:51.929 +00:00 [INF] - Plan execution started for deployment 9
<6> 2021-05-04 01:15:51.955 +00:00 [INF] - Executing command: "Command Group: (\n  [Create module edgeHub]\n  [Start module edgeHub]\n)"
<6> 2021-05-04 01:15:51.960 +00:00 [INF] - Executing command: "Create module edgeHub"
<6> 2021-05-04 01:15:53.228 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2021-05-04 01:15:54.355 +00:00 [INF] - Plan execution ended for deployment 9
<6> 2021-05-04 01:15:54.760 +00:00 [INF] - Updated reported properties
<6> 2021-05-04 01:15:59.979 +00:00 [INF] - Updated reported properties

edge-hub logs

<6> 2021-05-04 01:15:56.860 +00:00 [INF] - Version - 1.1.2.41361617 (5419edd44f82c229e3bb083cb52b76ce76b383b6)
<6> 2021-05-04 01:15:56.860 +00:00 [INF] - OptimizeForPerformance=True
<6> 2021-05-04 01:15:56.860 +00:00 [INF] - MessageAckTimeoutSecs=30
<6> 2021-05-04 01:15:56.861 +00:00 [INF] - Loaded server certificate with expiration date of "2021-07-22T23:38:05.0000000+00:00"
<6> 2021-05-04 01:15:56.871 +00:00 [INF] - Using Asp Net server for metrics
<6> 2021-05-04 01:15:56.920 +00:00 [INF] - Created new message store
<6> 2021-05-04 01:15:56.920 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2021-05-04 01:15:56.983 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2021-05-04 01:15:57.041 +00:00 [INF] - Initialized storing twin manager
<6> 2021-05-04 01:15:57.056 +00:00 [INF] - Initializing configuration
<6> 2021-05-04 01:15:57.075 +00:00 [INF] - New device connection for device TMIoTEdgeWin/$edgeHub
<6> 2021-05-04 01:15:57.084 +00:00 [INF] - Client TMIoTEdgeWin/$edgeHub connected to edgeHub, processing existing subscriptions.
<6> 2021-05-04 01:15:57.130 +00:00 [INF] - Processing pending subscriptions for TMIoTEdgeWin/$edgeHub
<4> 2021-05-04 01:15:57.348 +00:00 [WRN] - Empty edge hub configuration received. Ignoring...
<6> 2021-05-04 01:15:57.659 +00:00 [INF] - Attempting to connect to IoT Hub for client TMIoTEdgeWin/$edgeHub via AMQP...
<6> 2021-05-04 01:15:58.598 +00:00 [INF] - Exiting disconnected state
<6> 2021-05-04 01:15:58.611 +00:00 [INF] - Received device connected callback
<6> 2021-05-04 01:15:58.615 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2021-05-04 01:15:58.617 +00:00 [INF] - Processing subscriptions for client TMIoTEdgeWin/$edgeHub on device connected to cloud.
<6> 2021-05-04 01:15:58.617 +00:00 [INF] - Skipping TMIoTEdgeWin/$edgeHub for subscription processing, as it is currently being processed.
<6> 2021-05-04 01:15:58.625 +00:00 [INF] - Entering connected state
<6> 2021-05-04 01:15:58.629 +00:00 [INF] - Cloud connection for TMIoTEdgeWin/$edgeHub is True
<6> 2021-05-04 01:15:58.631 +00:00 [INF] - Connection status for TMIoTEdgeWin/$edgeHub changed to ConnectionEstablished
<6> 2021-05-04 01:15:58.632 +00:00 [INF] - Client TMIoTEdgeWin/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2021-05-04 01:15:58.632 +00:00 [INF] - Skipping TMIoTEdgeWin/$edgeHub for subscription processing, as it is currently being processed.
<6> 2021-05-04 01:15:58.634 +00:00 [INF] - Created cloud proxy for client TMIoTEdgeWin/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2021-05-04 01:15:58.637 +00:00 [INF] - Initialized cloud proxy 8b67d910-2477-41e4-8d88-a40b3d5f9775 for TMIoTEdgeWin/$edgeHub
<6> 2021-05-04 01:15:58.638 +00:00 [INF] - Created cloud connection for client TMIoTEdgeWin/$edgeHub
<6> 2021-05-04 01:15:58.867 +00:00 [INF] - Processing pending subscriptions for TMIoTEdgeWin/$edgeHub
<6> 2021-05-04 01:15:58.956 +00:00 [INF] - Updated reported properties for TMIoTEdgeWin/$edgeHub
<6> 2021-05-04 01:15:59.025 +00:00 [INF] - Created cloud endpoint iothub with max batch size 10 and fan-out factor of 10.
<6> 2021-05-04 01:15:59.063 +00:00 [INF] - Obtained edge hub config from module twin
<6> 2021-05-04 01:15:59.172 +00:00 [INF] - Set the following 1 route(s) in edge hub
<6> 2021-05-04 01:15:59.172 +00:00 [INF] - route: FROM /messages/* INTO $upstream
<6> 2021-05-04 01:15:59.174 +00:00 [INF] - Updated message store TTL to 7200 seconds
<6> 2021-05-04 01:15:59.174 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2021-05-04 01:15:59.186 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2021-05-04 01:15:59.187 +00:00 [INF] - Initialized edge hub configuration
<6> 2021-05-04 01:15:59.192 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<6> 2021-05-04 01:15:59.289 +00:00 [INF] - Scheduling server certificate renewal for "2021-07-22T23:35:35.0001077Z".
<6> 2021-05-04 01:15:59.291 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2021-05-04 01:15:59.296 +00:00 [INF] - Starting MQTT head
<6> 2021-05-04 01:15:59.328 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2021-05-04 01:15:59.393 +00:00 [INF] - Starting AMQP head
<6> 2021-05-04 01:15:59.397 +00:00 [INF] - Started MQTT head
<6> 2021-05-04 01:15:59.490 +00:00 [INF] - Started AMQP head
<6> 2021-05-04 01:15:59.491 +00:00 [INF] - Starting HTTP head
<4> 2021-05-04 01:15:59.650 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined in "UseKestrel()" instead.
<6> 2021-05-04 01:15:59.659 +00:00 [INF] - Started HTTP head
<6> 2021-05-04 01:16:02.250 +00:00 [INF] - Updated reported properties for TMIoTEdgeWin/$edgeHub
<6> 2021-05-04 01:20:59.209 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2021-05-04 01:25:59.201 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2021-05-04 01:30:59.195 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2021-05-04 01:35:59.208 +00:00 [INF] - Entering periodic task to reauthenticate connected clients

[ancaantochi]

Hi!

Why do you need to add the extra hosts config to edgeAgent?

It seems this is an issue with adding extra hosts in windows containers and the workaround it to set the IP and hostname in hosts file from Dockerfile (for custom modules).

Thanks!

[terrymandin]

My customer is deploying to an OT environment that does not have a DNS. They need to resolve the Azure IoT Hub URL: .azure-devices.net to the ExpressRoute endpoint.

Is your suggestion to pull the edgeAgent code and add the host name when creating the container?

[ancaantochi]

I think for edgeAgent is not easy to add to the dockerfile, so I was looking for a workaround. It worked for me to set the IP and hostname in the hosts file on the device and then it was able to resolve it from the container. Would this be a reasonable workaround for them?

[terrymandin]

Adding the ip and hostname to the device config would be a good workaround, but could not get that working. I just retested with the the following:

1. hostname/ip in the device hosts file (not the container hosts)
2. hostname/ip in the config.yaml (ExtraHosts)
3. hostname/ip in the configOptions in IoT Hub

I could not successfully resolve a "ping" from within the contianer (eg docker exec -it ...). I am interested why that works for you and not for me. I'd be happy to setup a Teams meeting if you would like to view my environment.

[ancaantochi]

I was able to reproduce the issue even when not using edge runtime, it looks like an issue with moby. I am still looking into finding a solution for it.

[github-actions]

This issue is being marked as stale because it has been open for 30 days with no activity.