Job with closed tunnel give error

[chris93111]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

Hello

i have 3 job terminating with error in same time, the job is cuted in output

a don't see a specific error to explain why the job termine with error , and we can see is canceled

Nov 1, 2022 @ 02:01:38.518 | cfg08-node-awx | XXXXXX{"eof": true} | worker
-- | -- | -- | --

  | Nov 1, 2022 @ 02:01:38.514 | cfg08-node-awx | {"zipfile": 3316} | worker

  | Nov 1, 2022 @ 02:01:38.512 | cfg08-node-awx | {"status": "canceled", "runner_ident": "676120"} | worker

  | Nov 1, 2022 @ 02:01:38.384 | cfg07-node-awx | 2022-11-01 01:01:38,384 DEBUG    [-] awx.main.commands.run_callback_receiver 30 is alive | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:38.384 | cfg07-node-awx | 2022-11-01 01:01:38,384 DEBUG    [-] awx.main.commands.run_callback_receiver 30 is alive | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:37.952 | cfg04-node-awx | XXXXX{"eof": true} | worker

  | Nov 1, 2022 @ 02:01:37.952 | cfg04-node-awx | XXXXX{"eof": true} | worker

  | Nov 1, 2022 @ 02:01:37.948 | cfg04-node-awx | {"zipfile": 3117} | worker

  | Nov 1, 2022 @ 02:01:37.948 | cfg04-node-awx | {"zipfile": 3117} | worker

  | Nov 1, 2022 @ 02:01:37.945 | cfg04-node-awx | {"status": "canceled", "runner_ident": "676095"} | worker

  | Nov 1, 2022 @ 02:01:37.945 | cfg04-node-awx | {"status": "canceled", "runner_ident": "676095"} | worker

  | Nov 1, 2022 @ 02:01:37.654 | cfg08-node-awx | {"zipfile": 3409} | worker

  | Nov 1, 2022 @ 02:01:37.654 | cfg08-node-awx | XXXXX{"eof": true} | worker

  | Nov 1, 2022 @ 02:01:37.650 | cfg08-node-awx | {"status": "canceled", "runner_ident": "676116"} | worker

  | Nov 1, 2022 @ 02:01:35.814 | cfg05-node-awx | 2022-11-01 01:01:35,813 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.scheduler recording task_manager metrics, last recorded 20.171624422073364 seconds ago | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:35.814 | cfg05-node-awx | 2022-11-01 01:01:35,814 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.scheduler Finishing task_manager Scheduler | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:35.777 | cfg05-node-awx | 2022-11-01 01:01:35,776 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.scheduler recording dependency_manager metrics, last recorded 20.17445182800293 seconds ago | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:35.777 | cfg05-node-awx | 2022-11-01 01:01:35,777 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.scheduler Finishing dependency_manager Scheduler | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:35.766 | cfg05-node-awx | 2022-11-01 01:01:35,766 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.scheduler Starting dependency_manager Scheduler | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:35.765 | cfg05-node-awx | 2022-11-01 01:01:35,765 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.scheduler Starting task_manager Scheduler | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:35.662 | cfg05-node-awx | 2022-11-01 01:01:35,661 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.dispatch task 4c23c6b6-631b-4798-a298-10e2ac1d19e2 starting awx.main.scheduler.tasks.dependency_manager(*[]) | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:35.662 | cfg05-node-awx | 2022-11-01 01:01:35,662 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.dispatch task 72d3c4d3-157d-498e-a1e3-e202d8d32852 starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*[]) | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:35.657 | cfg05-node-awx | 2022-11-01 01:01:35,656 DEBUG    [e19d1050366647769f5ce7ae0c82f52b] awx.main.dispatch task 2b4ca92e-8255-48ae-ad1e-50898211b821 starting awx.main.scheduler.tasks.task_manager(*[]) | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.625 | cfg07-node-awx | 2022-11-01 01:01:34,625 INFO     [44234efc25134bb78673d178a24523d9] awx.analytics.job_lifecycle job-676095 post run | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.625 | cfg07-node-awx | 2022-11-01 01:01:34,624 INFO     [7b73a7894a1f4b049ddcd716c71c04b4] awx.analytics.job_lifecycle job-676120 post run | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.625 | cfg07-node-awx | 2022-11-01 01:01:34,625 INFO     [44234efc25134bb78673d178a24523d9] awx.analytics.job_lifecycle job-676095 post run | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.625 | cfg07-node-awx | 2022-11-01 01:01:34,624 INFO     [7b73a7894a1f4b049ddcd716c71c04b4] awx.analytics.job_lifecycle job-676120 post run | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.625 | cfg07-node-awx | 2022-11-01 01:01:34,625 INFO     [44234efc25134bb78673d178a24523d9] awx.analytics.job_lifecycle job-676095 post run | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | 2022-11-01 01:01:34,622 DEBUG    [44234efc25134bb78673d178a24523d9] awx.main.tasks.jobs job 676095 (running) finished running, producing 189 events. | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | 2022-11-01 01:01:34,623 DEBUG    [7b73a7894a1f4b049ddcd716c71c04b4] awx.main.tasks.jobs job 676120 (running) finished running, producing 90 events. | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | DEBUG 2022/11/01 01:01:34 Control service closed | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | DEBUG 2022/11/01 01:01:34 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | 2022-11-01 01:01:34,622 DEBUG    [44234efc25134bb78673d178a24523d9] awx.main.tasks.jobs job 676095 (running) finished running, producing 189 events. | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | 2022-11-01 01:01:34,623 DEBUG    [7b73a7894a1f4b049ddcd716c71c04b4] awx.main.tasks.jobs job 676120 (running) finished running, producing 90 events. | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | DEBUG 2022/11/01 01:01:34 Control service closed | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | DEBUG 2022/11/01 01:01:34 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.623 | cfg07-node-awx | 2022-11-01 01:01:34,622 DEBUG    [44234efc25134bb78673d178a24523d9] awx.main.tasks.jobs job 676095 (running) finished running, producing 189 events. | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.622 | cfg07-node-awx | DEBUG 2022/11/01 01:01:34 Control service closed | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.622 | cfg07-node-awx | DEBUG 2022/11/01 01:01:34 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.622 | cfg07-node-awx | DEBUG 2022/11/01 01:01:34 Control service closed | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.622 | cfg07-node-awx | DEBUG 2022/11/01 01:01:34 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.601 | cfg06-node-awx | 2022-11-01 01:01:34,601 INFO     [d1aad01ea70846bb9e0a6e177a999fb1] awx.analytics.job_lifecycle job-676116 post run | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.599 | cfg06-node-awx | 2022-11-01 01:01:34,599 DEBUG    [d1aad01ea70846bb9e0a6e177a999fb1] awx.main.tasks.jobs job 676116 (running) finished running, producing 102 events. | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:34.599 | cfg06-node-awx | DEBUG 2022/11/01 01:01:34 Control service closed | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:34.599 | cfg06-node-awx | DEBUG 2022/11/01 01:01:34 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.855 | cfg06-node-awx | 2022-11-01 01:01:33,855 DEBUG    [0ab5e7fe472244b08f569c502e3e1be2] awx.main.tasks.system Last scheduler run was: 2022-11-01 01:01:33.826529+00:00 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.852 | cfg06-node-awx | 2022-11-01 01:01:33,851 DEBUG    [0ab5e7fe472244b08f569c502e3e1be2] awx.main.tasks.system Starting periodic scheduler | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.844 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.844 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.842 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: nlfXQbyu | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.842 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.842 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.842 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.842 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: nlfXQbyu | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.842 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.842 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.842 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.830 | cfg07-node-awx | 2022-11-01 01:01:33,829 DEBUG    [d59d0f4702f54984ba859720abfac05b] awx.main.tasks.system Last scheduler run was: 2022-11-01 01:01:25.652703+00:00 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.830 | cfg07-node-awx | 2022-11-01 01:01:33,829 DEBUG    [d59d0f4702f54984ba859720abfac05b] awx.main.tasks.system Last scheduler run was: 2022-11-01 01:01:25.652703+00:00 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.826 | cfg07-node-awx | 2022-11-01 01:01:33,826 DEBUG    [d59d0f4702f54984ba859720abfac05b] awx.main.tasks.system Starting periodic scheduler | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.826 | cfg07-node-awx | 2022-11-01 01:01:33,826 DEBUG    [d59d0f4702f54984ba859720abfac05b] awx.main.tasks.system Starting periodic scheduler | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.786 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.786 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.786 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.786 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.786 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.786 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.786 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.786 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.784 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: UgEYoOa2 | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.784 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: UgEYoOa2 | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.752 | cfg06-node-awx | 2022-11-01 01:01:33,750 DEBUG    [0ab5e7fe472244b08f569c502e3e1be2] awx.main.dispatch task 5ab21169-7d2f-4e9e-b0d4-0c2bab4e7d04 starting awx.main.tasks.system.awx_periodic_scheduler(*[]) | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.730 | cfg06-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.728 | cfg06-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: RUPbp9wZ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.728 | cfg06-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.728 | cfg06-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.728 | cfg06-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.722 | cfg07-node-awx | 2022-11-01 01:01:33,722 DEBUG    [d59d0f4702f54984ba859720abfac05b] awx.main.dispatch task 4cc2014f-26cf-41f8-8fc7-4f27fbe30f9d starting awx.main.tasks.system.awx_periodic_scheduler(*[]) | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.722 | cfg07-node-awx | 2022-11-01 01:01:33,722 DEBUG    [d59d0f4702f54984ba859720abfac05b] awx.main.dispatch task 4cc2014f-26cf-41f8-8fc7-4f27fbe30f9d starting awx.main.tasks.system.awx_periodic_scheduler(*[]) | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.591 | cfg07-node-awx | 2022-11-01 01:01:33,591 INFO     [44234efc25134bb78673d178a24523d9] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676095 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.591 | cfg07-node-awx | 2022-11-01 01:01:33,591 INFO     [44234efc25134bb78673d178a24523d9] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676095 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.591 | cfg07-node-awx | 2022-11-01 01:01:33,591 INFO     [44234efc25134bb78673d178a24523d9] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676095 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.591 | cfg07-node-awx | 2022-11-01 01:01:33,591 INFO     [44234efc25134bb78673d178a24523d9] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676095 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.591 | cfg07-node-awx | 2022-11-01 01:01:33,591 INFO     [44234efc25134bb78673d178a24523d9] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676095 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.591 | cfg07-node-awx | 2022-11-01 01:01:33,591 INFO     [44234efc25134bb78673d178a24523d9] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676095 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.590 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.590 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.589 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: nlfXQbyu | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.589 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.589 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.589 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.589 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: nlfXQbyu | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.589 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.589 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.589 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.542 | cfg07-node-awx | 2022-11-01 01:01:33,541 INFO     [7b73a7894a1f4b049ddcd716c71c04b4] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676120 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.542 | cfg07-node-awx | 2022-11-01 01:01:33,541 INFO     [7b73a7894a1f4b049ddcd716c71c04b4] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676120 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.542 | cfg07-node-awx | 2022-11-01 01:01:33,541 INFO     [7b73a7894a1f4b049ddcd716c71c04b4] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676120 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.542 | cfg07-node-awx | 2022-11-01 01:01:33,541 INFO     [7b73a7894a1f4b049ddcd716c71c04b4] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676120 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.532 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.532 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.531 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.531 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.530 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: UgEYoOa2 | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.530 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.530 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.530 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: UgEYoOa2 | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.530 | cfg07-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.530 | cfg07-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.482 | cfg06-node-awx | 2022-11-01 01:01:33,481 INFO     [d1aad01ea70846bb9e0a6e177a999fb1] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676116 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.482 | cfg06-node-awx | 2022-11-01 01:01:33,481 INFO     [d1aad01ea70846bb9e0a6e177a999fb1] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 676116 | awx-cfg-awx-task

  | Nov 1, 2022 @ 02:01:33.475 | cfg06-node-awx | DEBUG 2022/11/01 01:01:33 Client disconnected from control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.475 | cfg06-node-awx | WARNING 2022/11/01 01:01:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.475 | cfg06-node-awx | DEBUG 2022/11/01 01:01:33 Client connected to control service @ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.474 | cfg06-node-awx | DEBUG 2022/11/01 01:01:33 Stdout complete - closing channel for: RUPbp9wZ | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.474 | cfg06-node-awx | WARNING 2022/11/01 01:01:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection | awx-cfg-awx-ee

  | Nov 1, 2022 @ 02:01:33.389 | cfg04-node-awx | 2022-11-01 01:01:33,388 INFO: no action. I am (awx-cfg-awx-postgres-instance1-kcvw-0), the leader with the lock | database

  | Nov 1, 2022 @ 02:01:33.389 | cfg04-node-awx | 2022-11-01 01:01:33,388 INFO: no action. I am (awx-cfg-awx-postgres-instance1-kcvw-0), the leader with the lock | database

  | Nov 1, 2022 @ 02:01:30.743 | cfg07-node-awx | 2022-11-01 01:01:30,741 DEBUG    [40dd710def6e41c8a8e9cd084398b99e] awx.main.scheduler recording task_manager metrics, last recorded 20.054621696472168 seconds ago | awx-cfg-awx-task

But in logs of k3s i can see at the time of finished job the LB is updated and proxy tunnel is restarted

Nov  1 02:01:33 cfg04-node-awx k3s: time="2022-11-01T02:01:33+01:00" level=info msg="Updating load balancer server addresses -> [x.x.6.72:6443 x.x.6.79:6443]"
Nov  1 02:01:33 cfg04-node-awx k3s: time="2022-11-01T02:01:33+01:00" level=info msg="Stopped tunnel to x.x.6.87:6443"
Nov  1 02:01:33 cfg04-node-awx k3s: time="2022-11-01T02:01:33+01:00" level=info msg="Proxy done" err="context canceled" url="wss://x.x.6.87:6443/v1-k3s/connect"
Nov  1 02:01:33 cfg04-node-awx k3s: I1101 02:01:33.458431   44517 log.go:184] http: superfluous response.WriteHeader call from github.com/emicklei/go-restful.(*Response).WriteHeader (response.go:220)
Nov  1 02:01:34 cfg04-node-awx k3s: time="2022-11-01T02:01:34+01:00" level=info msg="Updating load balancer server addresses -> [x.x.6.87:6443 x.x.6.79:6443 x.x.6.72:6443]"
Nov  1 02:01:34 cfg04-node-awx k3s: time="2022-11-01T02:01:34+01:00" level=info msg="Connecting to proxy" url="wss://x.x.6.87:6443/v1-k3s/connect"

This can create perturbation for awx or receptor ?

4 awx controller
3 master and 8 nodes k8s cluster, 3 jobs error executed not on same node of awx controller

AWX version

21.6

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

2.12

Operating system

redhat 7.9

Web browser

Chrome

Steps to reproduce

restart k3s master with jobs running

Expected results

no error job

Actual results

job error

Additional information

Nobody has canceled theses jobs
restart the job work correctly
log-max-size is set to 500mb and log of this job is <500

they are other jobs in same interval with no error and same node

maybe same as #12297 #12677

in api job_explanation has empty

    "job_explanation": "Job terminated due to error",
    "execution_node": "",
    "controller_node": "awx-cfg-awx-b576ff56c-8tntv",
    "result_traceback": "XXXXXXX",
    "event_processing_finished": true,