Bad performance and unstable connections after upgrading from 3005.1 to 3006.6

[petervdb]

Description of Issue

salt-call state.apply runs significantly slower after upgrading the salt master and minions from 3005.1 to 3006.6.
Although most minions keep working we have a few minions which currently are not able to work properly with the master

Setup

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

[root@nagios4-test minion]# salt-call -l all grains.items
[TRACE ] Setting up log file logging: {'log_path': '/var/log/salt/minion', 'log_level': 'warning', 'log_format': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'root'}
[TRACE ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.

...

[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] Closing IPCMessageClient instance
[DEBUG ] Determining pillar cache
[DEBUG ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'nagios4-test.colo.elex.be', 'tcp://10.104.0.66:4506')
[TRACE ] ext_pillar_extra_data = {}
[TRACE ] Failed to send msg SaltReqTimeoutError('Message timed out')
[TRACE ] Failed to send msg SaltReqTimeoutError('Message timed out')
[TRACE ] Failed to send msg SaltReqTimeoutError('Message timed out')

Versions Report

Setup minion
salt-call --versions-report
Salt Version:
Salt: 3006.6

Python Version:
Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.14.2
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: centos 7.9.2009 Core
locale: utf-8
machine: x86_64
release: 3.10.0-1160.108.1.el7.x86_64
system: Linux
version: CentOS Linux 7.9.2009 Core

Setup master:

salt-master --versions-report
Salt Version:
Salt: 3006.6

Python Version:
Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
cffi: 1.16.0
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: 1.7.1
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.14.0
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.14.2
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: centos 7.9.2009 Core
locale: utf-8
machine: x86_64
release: 3.10.0-1160.81.1.el7.x86_64
system: Linux
version: CentOS Linux 7.9.2009 Core

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[petervdb]

Downgrading the minions to 3005.5-2 has solved my problem.

[dwoz]

@petervdb I think you may have run into an un-intentional change. Can you try setting request_channel_timeout: 60 in your minion's config on 3006.6?

[petervdb]

@petervdb I think you may have run into an un-intentional change. Can you try setting request_channel_timeout: 60 in your minion's config on 3006.6?

I currently get that timeout message after 1 minute. In debug mode you see it hanging for msome time before I see those timeout messages.
But, I will test it out, just to be sure.

[dwoz]

Seems related to #66061 like #66040 and #66071.

[petervdb]

Adding following setting to the minion config file:

request_channel_timeout: 60

Solved my issue with 3006.6.

[petervdb]

Adding following setting to the minion config file:

request_channel_timeout: 60

Solved my issue with 3006.6.