1. Build a custom Ops Agent that includes the fix and turn on the config ``` (cd submodules/fluent-bit && git fetch --tags && git checkout bf86a32dc59a7071493dddbdff29521797690233) git add submodules/fluent-bit git commit -m "Update fluent-bit submodule to bf86a32dc59a7071493dddbdff29521797690233." git checkout -b lingshi-ipv4 git push origin lingshi-ipv4 // Diff [OUTPUT] - Match_Regex ^(default_pipeline\.syslog)$ - Name stackdriver - Retry_Limit 3 - net.connect_timeout_log_error False - resource gce_instance - stackdriver_agent Google-Cloud-Ops-Agent-Logging/latest (BuildDistro=build_distro;Platform=linux;ShortName=linux_platform;ShortVersion=linux_platform_version) - tls On - tls.verify Off - workers 8 + Match_Regex ^(default_pipeline\.syslog)$ + Name stackdriver + Retry_Limit 3 + net.dns.prefer_ipv4 on + resource gce_instance + stackdriver_agent Google-Cloud-Ops-Agent-Logging/latest (BuildDistro=build_distro;Platform=linux;ShortName=linux_platform;ShortVersion=linux_platform_version) + tls On + tls.verify Off + workers 8 ``` 2. Create and connect to 2 Windows 2016 VMs. ``` gcloud compute instances create --image-project windows-cloud --image-family windows-2016 windows-2016-old-fix gcloud compute instances create --image-project windows-cloud --image-family windows-2016 windows-2016-new-fix ``` ### 3. Install Ops Agents On `windows-2016-old-fix`, install Ops Agent 2.8.1 (latest stable): ``` (New-Object Net.WebClient).DownloadFile("https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.ps1", "${env:UserProfile}\add-google-cloud-ops-agent-repo.ps1") cd "${env:UserProfile}" $env:REPO_SUFFIX="20220113-1"; .\add-google-cloud-ops-agent-repo.ps1 -AlsoInstall ``` On `windows-2016-new-fix`, install Ops Agent custom build: ``` (New-Object Net.WebClient).DownloadFile("https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.ps1", "${env:UserProfile}\add-google-cloud-ops-agent-repo.ps1") cd "${env:UserProfile}" $env:REPO_SUFFIX="20220113-2"; .\add-google-cloud-ops-agent-repo.ps1 -AlsoInstall ``` ### 4. Verify the logs do not have logging api connection errors Log file ``` notepad "C:\ProgramData\Google\Cloud Operations\Ops Agent\log\logging-module.log" ``` `windows-2016-old-fix` does not have them: ``` [2022/01/14 01:23:25] [ info] [engine] started (pid=4852) [2022/01/14 01:23:25] [ info] [storage] version=1.1.5, initializing... [2022/01/14 01:23:25] [ info] [storage] root path 'C:\ProgramData\Google\Cloud Operations\Ops Agent\run\buffers' [2022/01/14 01:23:25] [ info] [storage] normal synchronization mode, checksum enabled, max_chunks_up=128 [2022/01/14 01:23:25] [ info] [storage] backlog input plugin: storage_backlog.2 [2022/01/14 01:23:25] [ info] [cmetrics] version=0.2.2 [2022/01/14 01:23:25] [ info] [input:storage_backlog:storage_backlog.2] queue memory limit: 47.7M [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal [2022/01/14 01:23:25] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one [2022/01/14 01:23:25] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] metadata_server set to http://metadata.google.internal [2022/01/14 01:23:25] [ warn] [output:stackdriver:stackdriver.1] client_email is not defined, using a default one [2022/01/14 01:23:25] [ warn] [output:stackdriver:stackdriver.1] private_key is not defined, fetching it from metadata server [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] worker #0 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] worker #1 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] worker #2 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] worker #3 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] worker #4 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] worker #5 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] worker #6 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.0] worker #7 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] worker #0 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] worker #1 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] worker #2 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] worker #3 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] worker #4 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] worker #5 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] worker #6 started [2022/01/14 01:23:25] [ info] [output:stackdriver:stackdriver.1] worker #7 started [2022/01/14 01:23:25] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020 [2022/01/14 01:23:25] [ info] [sp] stream processor started ``` `windows-2016-new-fix` still have them: ``` [2022/01/14 00:44:44] [ info] [engine] started (pid=3716) [2022/01/14 00:44:44] [ info] [storage] version=1.1.5, initializing... [2022/01/14 00:44:44] [ info] [storage] root path 'C:\ProgramData\Google\Cloud Operations\Ops Agent\run\buffers' [2022/01/14 00:44:44] [ info] [storage] normal synchronization mode, checksum enabled, max_chunks_up=128 [2022/01/14 00:44:44] [ info] [storage] backlog input plugin: storage_backlog.2 [2022/01/14 00:44:44] [ info] [cmetrics] version=0.2.2 [2022/01/14 00:44:44] [ info] [input:storage_backlog:storage_backlog.2] queue memory limit: 47.7M [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal [2022/01/14 00:44:44] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one [2022/01/14 00:44:44] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] metadata_server set to http://metadata.google.internal [2022/01/14 00:44:44] [ warn] [output:stackdriver:stackdriver.1] client_email is not defined, using a default one [2022/01/14 00:44:44] [ warn] [output:stackdriver:stackdriver.1] private_key is not defined, fetching it from metadata server [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] worker #0 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] worker #1 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] worker #2 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] worker #3 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] worker #4 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] worker #5 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] worker #6 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.0] worker #7 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] worker #0 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] worker #1 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] worker #2 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] worker #3 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] worker #4 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] worker #5 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] worker #6 started [2022/01/14 00:44:44] [ info] [output:stackdriver:stackdriver.1] worker #7 started [2022/01/14 00:44:44] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020 [2022/01/14 00:44:44] [ info] [sp] stream processor started [2022/01/14 00:44:56] [error] [upstream] connection #1912 to logging.googleapis.com:443 timed out after 10 seconds [2022/01/14 00:44:57] [error] [upstream] connection #1856 to logging.googleapis.com:443 timed out after 10 seconds [2022/01/14 00:44:58] [error] [upstream] connection #1968 to logging.googleapis.com:443 timed out after 10 seconds [2022/01/14 00:44:59] [error] [upstream] connection #1984 to logging.googleapis.com:443 timed out after 10 seconds [2022/01/14 00:45:01] [error] [upstream] connection #2008 to logging.googleapis.com:443 timed out after 10 seconds [2022/01/14 00:45:02] [error] [upstream] connection #2044 to logging.googleapis.com:443 timed out after 10 seconds [2022/01/14 00:45:02] [error] [upstream] connection #2024 to logging.googleapis.com:443 timed out after 10 seconds [2022/01/14 00:45:03] [error] [upstream] connection #2072 to logging.googleapis.com:443 timed out after 10 seconds ```