Module cannot launch on yocto Hardknott Image

[wangxuefan123]

Expected Behavior

After edgeAgent launch, edgeHub and SimulatedTemperatureSensor should start shortly after.

Current Behavior

edgeAgent seems to run normally, but edgeHub and SimulatedTemperatureSensor failed to start.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.
1.Yocto Image compiled with meta-iotedge-1.1.3 and meta-rust-1.47 , meta-virtualization (Hardknott)
2. Start the iotedge on a custom board

Context (Environment)

Output of iotedge check

Click here

Configuration checks                                                                                                                                           
--------------------                                                                                                                                           
�√ config.yaml is well-formed - OK                                                                                                                             
�√ config.yaml has well-formed connection string - OK                                                                                                          
�√ container engine is installed and functional - OK                                                                                                           
�× config.yaml has correct hostname - Error                                                                                                                    
    config.yaml has hostname imx6qsabresd-sha001 but device reports hostname imx6qsabresd.                                                                     
    Hostname in config.yaml must either be identical to the device hostname or be a fully-qualified domain name that has the device hostname as the first comp.
�√ config.yaml has correct URIs for daemon mgmt endpoint - OK                                                                                                  
�√ latest security daemon - OK                                                                                                                                 
�‼ host time is close to real time - Warning                                                                                                                   
    Could not query NTP server                                                                                                                                 
�√ container time is close to host time - OK                                                                                                                   
�‼ DNS server - Warning                                                                                                                                        
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.                                                      
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.                                                                                   
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.                                                               
�‼ production readiness: certificates - Warning                                                                                                                
    The Edge device is using self-signed automatically-generated development certificates.                                                                     
    They will expire in 89 days (at 2021-09-30 00:00:22 UTC) causing module-to-module and downstream device communication to fail on an active deployment.     
    After the certs have expired, restarting the IoT Edge daemon will trigger it to generate new development certs.                                            
    Please consider using production certificates instead. See https://aka.ms/iotedge-prod-checklist-certs for best practices.                                 
�‼ production readiness: container engine - Warning                                                                                                            
    Device is not using a production-supported container engine (moby-engine).                                                                                 
    Please see https://aka.ms/iotedge-prod-checklist-moby for details.                                                                                         
�‼ production readiness: logs policy - Warning                                                                                                                 
    Container engine is not configured to rotate module logs which may cause it run out of disk space.                                                         
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.                                                                                  
    You can ignore this warning if you are setting log policy per module in the Edge deployment.                                                               
�‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning                                                          
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.                                                     
    Data might be lost if the module is deleted or updated.                                                                                                    
    Please see https://aka.ms/iotedge-storage-host for best practices.                                                                                         
�× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error                                                              
    Could not check current state of edgeHub container                                                                                                         
                                                                                                                                                               
Connectivity checks                                                                                                                                            
-------------------                                                                                                                                            
�× host can connect to and perform TLS handshake with IoT Hub AMQP port - Error                                                                                
    Could not connect to IOThubEdgeScale.azure-devices.net:5671                                                                                                
�√ host can connect to and perform TLS handshake with IoT Hub HTTPS / WebSockets port - OK                                                                     
�× host can connect to and perform TLS handshake with IoT Hub MQTT port - Error                                                                                
    Could not connect to IOThubEdgeScale.azure-devices.net:8883                                                                                                
�× container on the default network can connect to IoT Hub AMQP port - Error                                                                                   
    Container on the default network could not connect to IOThubEdgeScale.azure-devices.net:5671                                                               
�× container on the default network can connect to IoT Hub HTTPS / WebSockets port - Error                                                                     
    Container on the default network could not connect to IOThubEdgeScale.azure-devices.net:443                                                                
�× container on the default network can connect to IoT Hub MQTT port - Error                                                                                   
    Container on the default network could not connect to IOThubEdgeScale.azure-devices.net:8883                                                               
�× container on the IoT Edge module network can connect to IoT Hub AMQP port - Error                                                                           
    Container on the azure-iot-edge network could not connect to IOThubEdgeScale.azure-devices.net:5671                                                        
�√ container on the IoT Edge module network can connect to IoT Hub HTTPS / WebSockets port - OK                                                                
�× container on the IoT Edge module network can connect to IoT Hub MQTT port - Error                                                                           
    Container on the azure-iot-edge network could not connect to IOThubEdgeScale.azure-devices.net:8883                                                        
 

Device Information

• Host OS: Yocto 5.10.35 Hardknott:
• Architecture: armv7l
• Container OS: docker-ce

Runtime Versions

• aziot-edged [run iotedge version]: 1.1.3
• Edge Agent [image tag (e.g. 1.0.0)]: 1.1.3
• Edge Hub [image tag (e.g. 1.0.0)]: 1.1.3
• Docker/Moby : v19.03.15-ce, build 99e3ed8919

Logs

iotedge list

edgeAgent        running          Up 29 minutes    mcr.microsoft.com/azureiotedge-agent:1.1   

edge-agent logs

2021-07-02 13:26:13 +00:00 Starting Edge Agent                                                                                                                 
2021-07-02 13:26:13 +00:00 Changing ownership of management socket: /var/run/iotedge/mgmt.sock                                                                 
2021-07-02 13:26:15.151 +00:00 Edge Agent Main()                                                                                                               
<6> 2021-07-02 13:26:17.348 +00:00 [INF] - Initializing Edge Agent.                                                                                            
<6> 2021-07-02 13:26:18.851 +00:00 [INF] - Version - 1.1.3.42832849 (68e71a5384b14956775241557877497274d0ce7e)                                                 
<6> 2021-07-02 13:26:18.855 +00:00 [INF] -                                                                                                                     
        �█�█�█�█�█�╗ �█�█�█�█�█�█�█�╗�█�█�╗   �█�█�╗�█�█�█�█�█�█�╗ �█�█�█�█�█�█�█�╗                                                                            
       �█�█�╔�═�═�█�█�╗�╚�═�═�█�█�█�╔�╝�█�█�║   �█�█�║�█�█�╔�═�═�█�█�╗�█�█�╔�═�═�═�═�╝                                                                         
       �█�█�█�█�█�█�█�║  �█�█�█�╔�╝ �█�█�║   �█�█�║�█�█�█�█�█�█�╔�╝�█�█�█�█�█�╗                                                                                
       �█�█�╔�═�═�█�█�║ �█�█�█�╔�╝  �█�█�║   �█�█�║�█�█�╔�═�═�█�█�╗�█�█�╔�═�═�╝                                                                                
       �█�█�║  �█�█�║�█�█�█�█�█�█�█�╗�╚�█�█�█�█�█�█�╔�╝�█�█�║  �█�█�║�█�█�█�█�█�█�█�╗                                                                          
       �╚�═�╝  �╚�═�╝�╚�═�═�═�═�═�═�╝ �╚�═�═�═�═�═�╝ �╚�═�╝  �╚�═�╝�╚�═�═�═�═�═�═�╝                                                                            
                                                                                                                                                               
 �█�█�╗ �█�█�█�█�█�█�╗ �█�█�█�█�█�█�█�█�╗    �█�█�█�█�█�█�█�╗�█�█�█�█�█�█�╗  �█�█�█�█�█�█�╗ �█�█�█�█�█�█�█�╗                                                   
 �█�█�║�█�█�╔�═�═�═�█�█�╗�╚�═�═�█�█�╔�═�═�╝    �█�█�╔�═�═�═�═�╝�█�█�╔�═�═�█�█�╗�█�█�╔�═�═�═�═�╝ �█�█�╔�═�═�                                                   ╝
 �█�█�║�█�█�║   �█�█�║   �█�█�║       �█�█�█�█�█�╗  �█�█�║  �█�█�║�█�█�║  �█�█�█�╗�█�█�█�█�█�╗                                                                 
 �█�█�║�█�█�║   �█�█�║   �█�█�║       �█�█�╔�═�═�╝  �█�█�║  �█�█�║�█�█�║   �█�█�║�█�█�╔�═�═�╝                                                                  
 �█�█�║�╚�█�█�█�█�█�█�╔�╝   �█�█�║       �█�█�█�█�█�█�█�╗�█�█�█�█�█�█�╔�╝�╚�█�█�█�█�█�█�╔�╝�█�█�█�█�█�█�█�╗                                                    
 �╚�═�╝ �╚�═�═�═�═�═�╝    �╚�═�╝       �╚�═�═�═�═�═�═�╝�╚�═�═�═�═�═�╝  �╚�═�═�═�═�═�╝ �╚�═�═�═�═�═�═�╝                                                         
                                                                                                                                                               
<6> 2021-07-02 13:26:19.155 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}                            
<6> 2021-07-02 13:26:20.123 +00:00 [INF] - Installing certificates [CN=Test Edge Owner CA:09/30/2021 00:00:22] to Root                                         
<6> 2021-07-02 13:26:21.030 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: /metrics                                                  
<6> 2021-07-02 13:26:22.127 +00:00 [INF] - Updating performance metrics every 05m:00s                                                                          
<6> 2021-07-02 13:26:22.155 +00:00 [INF] - Started operation Get system resources                                                                              
<6> 2021-07-02 13:26:22.165 +00:00 [INF] - Collecting metadata metrics                                                                                         
<6> 2021-07-02 13:26:22.557 +00:00 [INF] - Set metadata metrics: 1.1.3.42832849 (68e71a5384b14956775241557877497274d0ce7e), {"Enabled":false,"DisableCloudSubse
<6> 2021-07-02 13:26:22.716 +00:00 [INF] - Started operation Checkpoint Availability                                                                           
<6> 2021-07-02 13:26:22.768 +00:00 [INF] - Started operation refresh twin config                                                                               
<6> 2021-07-02 13:26:22.880 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...                                                    
<6> 2021-07-02 13:26:25.156 +00:00 [INF] - Created persistent store at /tmp/edgeAgent                                                                          
<6> 2021-07-02 13:26:25.425 +00:00 [INF] - Started operation Metrics Scrape                                                                                    
<6> 2021-07-02 13:26:25.425 +00:00 [INF] - Started operation Metrics Upload                                                                                    
Scraping frequency: 01:00:00                                                                                                                                   
Upload Frequency: 1.00:00:00                                                                                                                                   
<6> 2021-07-02 13:26:26.883 +00:00 [INF] - Registering request handler UploadModuleLogs                                                                        
<6> 2021-07-02 13:26:26.884 +00:00 [INF] - Registering request handler GetModuleLogs                                                                           
<6> 2021-07-02 13:26:26.885 +00:00 [INF] - Registering request handler UploadSupportBundle                                                                     
<6> 2021-07-02 13:26:26.886 +00:00 [INF] - Registering request handler RestartModule                                                                           
<4> 2021-07-02 13:26:32.254 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<6> 2021-07-02 13:26:32.270 +00:00 [INF] - Edge agent config backup file does not exist - /tmp/edgeAgent/backup.json                                           
                                                                                                                                                               
<6> 2021-07-02 13:26:36.835 +00:00 [INF] - Termination requested, initiating shutdown.                                                                         
<6> 2021-07-02 13:26:36.840 +00:00 [INF] - Main thread terminated                                                                                              
<6> 2021-07-02 13:26:36.849 +00:00 [INF] - Initiating shutdown cleanup.                                                                                        
<6> 2021-07-02 13:26:36.857 +00:00 [INF] - Stopping all modules...                                                                                             
<6> 2021-07-02 13:26:36.885 +00:00 [INF] - Waiting for cleanup to finish                                                                                       
<6> 2021-07-02 13:26:36.991 +00:00 [INF] - Plan execution started for deployment -1                                                                            
<6> 2021-07-02 13:26:37.004 +00:00 [INF] - Plan execution ended for deployment -1                                                                              
<6> 2021-07-02 13:26:37.005 +00:00 [INF] - Completed stopping all modules.                                                                                     
<6> 2021-07-02 13:26:37.009 +00:00 [INF] - Completed shutdown cleanup.                                                                                         
<6> 2021-07-02 13:26:37.017 +00:00 [INF] - Done with cleanup. Shutting down.                                                                                   
Session terminated, terminating shell... ...terminated.                                                                                                        
2021-07-02 13:26:39 +00:00 Starting Edge Agent                                                                                                                 
2021-07-02 13:26:39 +00:00 Changing ownership of storage folder: /tmp/edgeAgent to 1000                                                                        
2021-07-02 13:26:39 +00:00 Changing ownership of management socket: /var/run/iotedge/mgmt.sock                                                                 
2021-07-02 13:26:40.338 +00:00 Edge Agent Main()                                                                                                               
<6> 2021-07-02 13:26:41.433 +00:00 [INF] - Initializing Edge Agent.                                                                                            
<6> 2021-07-02 13:26:42.375 +00:00 [INF] - Version - 1.1.3.42832849 (68e71a5384b14956775241557877497274d0ce7e)                                                 
<6> 2021-07-02 13:26:42.378 +00:00 [INF] -                                                                                                                     
        �█�█�█�█�█�╗ �█�█�█�█�█�█�█�╗�█�█�╗   �█�█�╗�█�█�█�█�█�█�╗ �█�█�█�█�█�█�█�╗                                                                            
       �█�█�╔�═�═�█�█�╗�╚�═�═�█�█�█�╔�╝�█�█�║   �█�█�║�█�█�╔�═�═�█�█�╗�█�█�╔�═�═�═�═�╝                                                                         
       �█�█�█�█�█�█�█�║  �█�█�█�╔�╝ �█�█�║   �█�█�║�█�█�█�█�█�█�╔�╝�█�█�█�█�█�╗                                                                                
       �█�█�╔�═�═�█�█�║ �█�█�█�╔�╝  �█�█�║   �█�█�║�█�█�╔�═�═�█�█�╗�█�█�╔�═�═�╝                                                                                
       �█�█�║  �█�█�║�█�█�█�█�█�█�█�╗�╚�█�█�█�█�█�█�╔�╝�█�█�║  �█�█�║�█�█�█�█�█�█�█�╗                                                                          
       �╚�═�╝  �╚�═�╝�╚�═�═�═�═�═�═�╝ �╚�═�═�═�═�═�╝ �╚�═�╝  �╚�═�╝�╚�═�═�═�═�═�═�╝                                                                            
                                                                                                                                                               
 �█�█�╗ �█�█�█�█�█�█�╗ �█�█�█�█�█�█�█�█�╗    �█�█�█�█�█�█�█�╗�█�█�█�█�█�█�╗  �█�█�█�█�█�█�╗ �█�█�█�█�█�█�█�╗                                                   
 �█�█�║�█�█�╔�═�═�═�█�█�╗�╚�═�═�█�█�╔�═�═�╝    �█�█�╔�═�═�═�═�╝�█�█�╔�═�═�█�█�╗�█�█�╔�═�═�═�═�╝ �█�█�╔�═�═�                                                   ╝
 �█�█�║�█�█�║   �█�█�║   �█�█�║       �█�█�█�█�█�╗  �█�█�║  �█�█�║�█�█�║  �█�█�█�╗�█�█�█�█�█�╗                                                                 
 �█�█�║�█�█�║   �█�█�║   �█�█�║       �█�█�╔�═�═�╝  �█�█�║  �█�█�║�█�█�║   �█�█�║�█�█�╔�═�═�╝                                                                  
 �█�█�║�╚�█�█�█�█�█�█�╔�╝   �█�█�║       �█�█�█�█�█�█�█�╗�█�█�█�█�█�█�╔�╝�╚�█�█�█�█�█�█�╔�╝�█�█�█�█�█�█�█�╗                                                    
 �╚�═�╝ �╚�═�═�═�═�═�╝    �╚�═�╝       �╚�═�═�═�═�═�═�╝�╚�═�═�═�═�═�╝  �╚�═�═�═�═�═�╝ �╚�═�═�═�═�═�═�╝                                                         
                                                                                                                                                               
<6> 2021-07-02 13:26:42.628 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}                            
<6> 2021-07-02 13:26:43.230 +00:00 [INF] - Installing certificates [CN=Test Edge Owner CA:09/30/2021 00:00:22] to Root                                         
<6> 2021-07-02 13:26:43.968 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: /metrics                                                  
<6> 2021-07-02 13:26:44.813 +00:00 [INF] - Updating performance metrics every 05m:00s                                                                          
<6> 2021-07-02 13:26:44.840 +00:00 [INF] - Started operation Get system resources                                                                              
<6> 2021-07-02 13:26:44.849 +00:00 [INF] - Collecting metadata metrics                                                                                         
<6> 2021-07-02 13:26:45.227 +00:00 [INF] - Set metadata metrics: 1.1.3.42832849 (68e71a5384b14956775241557877497274d0ce7e), {"Enabled":false,"DisableCloudSubse
<6> 2021-07-02 13:26:45.323 +00:00 [INF] - Started operation Checkpoint Availability                                                                           
<6> 2021-07-02 13:26:45.348 +00:00 [INF] - Started operation refresh twin config                                                                               
<6> 2021-07-02 13:26:45.437 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...                                                    
<6> 2021-07-02 13:26:47.169 +00:00 [INF] - Created persistent store at /tmp/edgeAgent                                                                          
<6> 2021-07-02 13:26:47.384 +00:00 [INF] - Started operation Metrics Scrape                                                                                    
<6> 2021-07-02 13:26:47.385 +00:00 [INF] - Started operation Metrics Upload                                                                                    
Scraping frequency: 01:00:00                                                                                                                                   
Upload Frequency: 1.00:00:00                                                                                                                                   
<6> 2021-07-02 13:26:48.682 +00:00 [INF] - Registering request handler UploadModuleLogs                                                                        
<6> 2021-07-02 13:26:48.684 +00:00 [INF] - Registering request handler GetModuleLogs                                                                           
<6> 2021-07-02 13:26:48.685 +00:00 [INF] - Registering request handler UploadSupportBundle                                                                     
<6> 2021-07-02 13:26:48.685 +00:00 [INF] - Registering request handler RestartModule                                                                           
<4> 2021-07-02 13:26:54.045 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<6> 2021-07-02 13:26:54.061 +00:00 [INF] - Edge agent config backup file does not exist - /tmp/edgeAgent/backup.json                                           
<4> 2021-07-02 13:27:04.105 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:27:14.139 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<6> 2021-07-02 13:27:16.644 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...                                              
<4> 2021-07-02 13:27:24.160 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:27:34.185 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:27:44.217 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:27:47.511 +00:00 [WRN] - Error creating a device-to-cloud connection                                                                         
System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.) (Transient network error occurred, please retry.)    
 ---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.                                  
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable                                                                                
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)                                                      
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)                                                     
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)                                               
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)                                                           
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)                                                                    
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)                                        
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)                                 
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()                  
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   --- End of inner exception stack trace ---                                                                                                                  
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()                              
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)                                   
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()                                                                                                
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.CreateAndOpenSdkModuleClient(UpstreamProtocol upstreamProtocol, ConnectionStatusChangesHa7
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.<>c__DisplayClass19_1.<<CreateSdkModuleClient>b__2>d.MoveNext() in /home/vsts/work/1/s/ed3
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Edge.Util.Fallback.ExecuteAsync[T](Func`1[] options)                                                                             
   --- End of inner exception stack trace ---                                                                                                                  
 ---> (Inner Exception #1) Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.             
 ---> System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server                                                            
 ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable                                                                                   
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable                                                                                
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)                                                 
   --- End of inner exception stack trace ---                                                                                                                  
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)                                                 
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)                     
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)                          
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)                               
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)            
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)                                               
   at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)                     
   at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)                     
   at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)                                                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.CreateClientWebSocketAsync(Uri websocketUri, TimeSpan timeout)                         
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.CreateClientWebSocketTransportAsync(TimeSpan timeout)                                  
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)                                                      
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)                                                     
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)                                               
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)                                                           
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)                                                                    
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)                                        
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)                                 
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()                  
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   --- End of inner exception stack trace ---                                                                                                                  
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()                              
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)                                   
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()                                                                                                
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.CreateAndOpenSdkModuleClient(UpstreamProtocol upstreamProtocol, ConnectionStatusChangesHa7
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.<>c__DisplayClass19_1.<<CreateSdkModuleClient>b__3>d.MoveNext() in /home/vsts/work/1/s/ed9
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Edge.Util.Fallback.ExecuteAsync[T](Func`1[] options)<---                                                                         
<6> 2021-07-02 13:27:47.713 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...                                                    
<4> 2021-07-02 13:27:54.237 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:28:04.252 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:28:14.258 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<6> 2021-07-02 13:28:18.231 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...                                              
<4> 2021-07-02 13:28:24.281 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:28:34.286 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:28:44.300 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:28:48.902 +00:00 [WRN] - Error creating a device-to-cloud connection                                                                         
System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.) (Transient network error occurred, please retry.)    
 ---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.                                  
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable                                                                                
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)                                                      
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)                                                     
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)                                               
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)                                                           
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)                                                                    
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)                                        
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)                                 
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()                  
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   --- End of inner exception stack trace ---                                                                                                                  
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()                              
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)                                   
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()                                                                                                
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.CreateAndOpenSdkModuleClient(UpstreamProtocol upstreamProtocol, ConnectionStatusChangesHa7
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.<>c__DisplayClass19_1.<<CreateSdkModuleClient>b__2>d.MoveNext() in /home/vsts/work/1/s/ed3
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Edge.Util.Fallback.ExecuteAsync[T](Func`1[] options)                                                                             
   --- End of inner exception stack trace ---                                                                                                                  
 ---> (Inner Exception #1) Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.             
 ---> System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server                                                            
 ---> System.Net.Http.HttpRequestException: Resource temporarily unavailable                                                                                   
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable                                                                                
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)                                                 
   --- End of inner exception stack trace ---                                                                                                                  
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)                                                 
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)                     
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)                          
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)                               
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)            
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)                                               
   at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)                     
   at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)                     
   at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)                                                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.CreateClientWebSocketAsync(Uri websocketUri, TimeSpan timeout)                         
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.CreateClientWebSocketTransportAsync(TimeSpan timeout)                                  
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)                                                      
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)                                                     
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)                                               
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)                                                           
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)                                                                    
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)                                        
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)                                 
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()                  
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   --- End of inner exception stack trace ---                                                                                                                  
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()                              
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)                                   
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()                                                                                                
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.CreateAndOpenSdkModuleClient(UpstreamProtocol upstreamProtocol, ConnectionStatusChangesHa7
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.<>c__DisplayClass19_1.<<CreateSdkModuleClient>b__3>d.MoveNext() in /home/vsts/work/1/s/ed9
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Edge.Util.Fallback.ExecuteAsync[T](Func`1[] options)<---                                                                         
<4> 2021-07-02 13:28:54.314 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<6> 2021-07-02 13:28:55.146 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...                                                    
<4> 2021-07-02 13:29:04.314 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:29:14.319 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:29:24.328 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<6> 2021-07-02 13:29:25.665 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...                                              
<4> 2021-07-02 13:29:34.342 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:29:44.362 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:29:54.370 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:29:56.280 +00:00 [WRN] - Error creating a device-to-cloud connection                                                                         
System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.) (Transient network error occurred, please retry.)    
 ---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.                                  
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable                                                                                
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)                                                      
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)                                                     
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)                                               
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)                                                           
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)                                                                    
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)                                        
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)                                 
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()                  
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   --- End of inner exception stack trace ---                                                                                                                  
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()                              
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)                                   
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()                                                                                                
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.CreateAndOpenSdkModuleClient(UpstreamProtocol upstreamProtocol, ConnectionStatusChangesHa7
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.<>c__DisplayClass19_1.<<CreateSdkModuleClient>b__2>d.MoveNext() in /home/vsts/work/1/s/ed3
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Edge.Util.Fallback.ExecuteAsync[T](Func`1[] options)                                                                             
   --- End of inner exception stack trace ---                                                                                                                  
 ---> (Inner Exception #1) Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.             
 ---> System.Net.WebSockets.WebSocketException (1806782968): Unable to connect to the remote server ---> System.Net.Http.HttpRequestException: Resource tempore
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable                                                                                
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)                                                 
   --- End of inner exception stack trace ---                                                                                                                  
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)                                                 
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)                     
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)                          
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)                               
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)            
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)                                               
   at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)                     
   at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)                     
   at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)                                                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.CreateClientWebSocketAsync(Uri websocketUri, TimeSpan timeout)                         
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.CreateClientWebSocketTransportAsync(TimeSpan timeout)                                  
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)                                                      
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)                                                     
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)                                               
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)                     
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)                                                           
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)                                                                    
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)                                        
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)                                 
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()                  
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   --- End of inner exception stack trace ---                                                                                                                  
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                  
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()                              
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)                                   
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()                                                                                                
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.CreateAndOpenSdkModuleClient(UpstreamProtocol upstreamProtocol, ConnectionStatusChangesHa7
   at Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider.<>c__DisplayClass19_1.<<CreateSdkModuleClient>b__3>d.MoveNext() in /home/vsts/work/1/s/ed9
--- End of stack trace from previous location where exception was thrown ---                                                                                   
   at Microsoft.Azure.Devices.Edge.Util.Fallback.ExecuteAsync[T](Func`1[] options)<---                                                                         
<4> 2021-07-02 13:30:04.377 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<6> 2021-07-02 13:30:08.199 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...                                                    
<4> 2021-07-02 13:30:14.395 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:30:24.407 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<4> 2021-07-02 13:30:34.431 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead    
<6> 2021-07-02 13:30:38.727 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...  

edge-hub logs

Jul 02 13:41:38 imx6qsabresd iotedged[750]: 2021-07-02T13:41:38Z [INFO] - Checking edge runtime status
Jul 02 13:41:38 imx6qsabresd iotedged[750]: 2021-07-02T13:41:38Z [INFO] - Edge runtime is running.
Jul 02 13:41:39 imx6qsabresd iotedged[750]: 2021-07-02T13:41:39Z [INFO] - [mgmt] - - - [2021-07-02 13:41:39.891073778 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 609 "-" "-" auth_id(-)
Jul 02 13:41:49 imx6qsabresd iotedged[750]: 2021-07-02T13:41:49Z [INFO] - [mgmt] - - - [2021-07-02 13:41:49.905713113 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 609 "-" "-" auth_id(-)
Jul 02 13:41:59 imx6qsabresd iotedged[750]: 2021-07-02T13:41:59Z [INFO] - [mgmt] - - - [2021-07-02 13:41:59.903116114 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 609 "-" "-" auth_id(-)
Jul 02 13:42:09 imx6qsabresd iotedged[750]: 2021-07-02T13:42:09Z [INFO] - [mgmt] - - - [2021-07-02 13:42:09.926355448 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 609 "-" "-" auth_id(-)
Jul 02 13:42:19 imx6qsabresd iotedged[750]: 2021-07-02T13:42:19Z [INFO] - [mgmt] - - - [2021-07-02 13:42:19.932128116 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 609 "-" "-" auth_id(-)
Jul 02 13:42:28 imx6qsabresd iotedged[750]: 2021-07-02T13:42:28Z [INFO] - Getting logs for module edgeHub...
Jul 02 13:42:28 imx6qsabresd iotedged[750]: 2021-07-02T13:42:28Z [WARN] - Could not get logs for module edgeHub
Jul 02 13:42:28 imx6qsabresd iotedged[750]: 2021-07-02T13:42:28Z [WARN] -         caused by: Container runtime error - Api(ApiError { code: 404, content: None })
Jul 02 13:42:28 imx6qsabresd iotedged[750]: 2021-07-02T13:42:28Z [INFO] - [mgmt] - - - [2021-07-02 13:42:28.748926451 UTC] "GET /modules/edgeHub/logs?api-version=2020-07-07&follow=false&tail=all&timestamps=false&since=1625146948 HTTP/1.1" 500 Internal Server Error 186 "-" "iotedge/0.1.0" auth_id(-)
Jul 02 13:42:29 imx6qsabresd iotedged[750]: 2021-07-02T13:42:29Z [INFO] - [mgmt] - - - [2021-07-02 13:42:29.922291451 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 609 "-" "-" auth_id(-)
Jul 02 13:42:38 imx6qsabresd iotedged[750]: 2021-07-02T13:42:38Z [INFO] - Checking edge runtime status
Jul 02 13:42:38 imx6qsabresd iotedged[750]: 2021-07-02T13:42:38Z [INFO] - Edge runtime is running.
Jul 02 13:42:39 imx6qsabresd iotedged[750]: 2021-07-02T13:42:39Z [INFO] - [mgmt] - - - [2021-07-02 13:42:39.924093119 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 609 "-" "-" auth_id(-)
Jul 02 13:42:40 imx6qsabresd iotedged[750]: 2021-07-02T13:42:40Z [INFO] - Getting logs for module edgeAgent...
Jul 02 13:42:40 imx6qsabresd iotedged[750]: 2021-07-02T13:42:40Z [INFO] - Successfully got logs for module edgeAgent
Jul 02 13:42:40 imx6qsabresd iotedged[750]: 2021-07-02T13:42:40Z [INFO] - [mgmt] - - - [2021-07-02 13:42:40.313012452 UTC] "GET /modules/edgeAgent/logs?api-version=2020-07-07&follow=false&tail=all&timestamps=false&since=1625146960 HTTP/1.1" 200 OK - "-" "iotedge/0.1.0" auth_id(-)

Additional Information

Please provide any additional information that may be helpful in understanding the issue.
support_bundle.zip

[wangxuefan123]

It turns out after setting time ,run

hwclock -w

then edgeHub is able to start, but still meet problem with Simulated TemperatureSensor log

iotedge logs SimulatedTemperatureSensor

[2021-07-02 16:47:11 +00:00]: Starting Module                                                                                                                                                              
SimulatedTemperatureSensor Main() started.                                                                                                                                                                 
Initializing simulated temperature sensor to send 500 messages, at an interval of 5 seconds.                                                                                                               
To change this, set the environment variable MessageCount to the number of messages that should be sent (set it to -1 to send unlimited messages).                                                         
[Information]: Trying to initialize module client using transport type [Amqp_Tcp_Only].                                                                                                                    
Unhandled exception. System.AggregateException: One or more errors occurred. (Transient network error occurred, please retry.)                                                                             
 ---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.                                                                              
 ---> System.Net.Sockets.SocketException (113): No route to host                                                                                                                                           
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpIoTTransport.InitializeAsync(TimeSpan timeout)                                                                                                  
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIoTConnector.OpenConnectionAsync(TimeSpan timeout)                                                                                                 
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(TimeSpan timeout)                                                                                           
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(DeviceIdentity deviceIdentity, TimeSpan timeout)                                                                 
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.EnsureSessionAsync(TimeSpan timeout)                                                                                                       
   at Microsoft.Azure.Devices.Client.Transport.AmqpIoT.AmqpUnit.OpenAsync(TimeSpan timeout)                                                                                                                
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)                                                                                    
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)                                                                             
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass23_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()                                                              
--- End of stack trace from previous location where exception was thrown ---                                                                                                                               
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                                                              
   --- End of inner exception stack trace ---                                                                                                                                                              
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)                                                                              
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass33_0.<<OpenInternalAsync>b__0>d.MoveNext()                                                                          
--- End of stack trace from previous location where exception was thrown ---                                                                                                                               
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)                                                                               
   at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()                                                                                                                                            
   at SimulatedTemperatureSensor.Program.<>c__DisplayClass18_0.<<CreateModuleClientAsync>b__1>d.MoveNext() in /home/vsts/work/1/s/edge-modules/SimulatedTemperatureSensor/src/Program.cs:line 294          
--- End of stack trace from previous location where exception was thrown ---                                                                                                                               
   at SimulatedTemperatureSensor.Program.CreateModuleClientAsync(TransportType transportType, ITransientErrorDetectionStrategy transientErrorDetectionStrategy, RetryStrategy retryStrategy) in /home/vsts2
   at SimulatedTemperatureSensor.Program.MainAsync() in /home/vsts/work/1/s/edge-modules/SimulatedTemperatureSensor/src/Program.cs:line 79                                                                 
   --- End of inner exception stack trace ---                                                                                                                                                              
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)                                                                                                                      
   at System.Threading.Tasks.Task`1.get_Result()                                                                                                                                                           
   at SimulatedTemperatureSensor.Program.Main() in /home/vsts/work/1/s/edge-modules/SimulatedTemperatureSensor/src/Program.cs:line 48    

[vipeller]

@wangxuefan123 if you wait until you see edgeHub running and then start SimulatedTemperatureSensor (e.g. using the command 'iotedge restart SimulatedTemperatureSensor', you still see the 'no route to host' message?

[wangxuefan123]

@wangxuefan123 if you wait until you see edgeHub running and then start SimulatedTemperatureSensor (e.g. using the command 'iotedge restart SimulatedTemperatureSensor', you still see the 'no route to host' message?

Yes, I've tried restart SimulatedTemperatureSensor several times but it just keeps repeating the 'no route to host' message.

[github-actions]

This issue is being marked as stale because it has been open for 30 days with no activity.