installed 3007 TypeError: object NoneType can't be used in 'await' expression #66177
Comments
|
Can you try stopping your Salt master and minion, then run |
|
@doesitblend , thanks for the oneliner script to cleanup the .pyc. salt-minion[1567838]: [ERROR ] Publish server binding pub to /var/run/salt/minion/minion_event_0f4efbad65_pub.ipc ssl=None |
|
Tried the one liner and now the minion reports: 2024-03-07 22:32:33,860 [salt.transport.zeromq:396 ][ERROR ][1756164] Exception while running callback The master reports (lots and lots of these): File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 456, in salt_run File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/run.py", line 34, in run File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/runner.py", line 303, in run File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/mixins.py", line 546, in _proc_function File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/mixins.py", line 383, in low File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in call File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1233, in run File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1248, in _run_as File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/runners/state.py", line 310, in event File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in call File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1233, in run File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1248, in _run_as File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 2576, in event File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 127, in get_event File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 928, in init File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 265, in init File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 77, in init File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 219, in init File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in init |
|
@z900collector , the one-liner worked for me to remove *.pyc files and I don't have your message above. |
|
After running the oneliner on both master and minion, it is working again. But it does not prevent the minion from returning properly. |
|
I'm facing the same issue, after deleting .pyc files the minions get online. However these error lines still flood the minion logs. |
|
rocky9t01a test VM upgraded from 3006.6 to 3007.0.
[me@rocky9t01 salt]$
sudo salt-minion -V
Salt Version:
Salt: 3007.0
Python Version: Dependency Versions: Salt Package Information: System Versions: [me@rocky9t01 salt]$ |
This worked, but only for two days, errors returned on three minions so far. |
|
Hi all. I have the same issues with saltminion 3007.0-0 2024-03-12 10:27:41,901 [salt.transport.tcp:1410][ERROR ][16652] Publish server binding pub to 127.0.0.1:4510 ssl=None Both on Linux as on windows |
|
Hi I see there are comments that this is a 'log flood' issue. I think it's a bit more than that - my minions disconnect from the master. |
|
This issue has been opened for two weeks and it is still not gaining enough attention to be classified as bug. |
|
Hi, my minions seem to become slow to respond and commands seem to have timeouts. When I delete the .pyc files all is fast and happy. A few days later things go slow and time out again. Log when minion does not respond, doing basic test.ping: |
|
I've reinstall 3006.7 on all my minions but kept the Salt Master at 3007.0
until this is fixed.
Sid
…On Tue, 19 Mar 2024, 23:05 SwimGeek, ***@***.***> wrote:
Hi, my minions seem to become slow to respond and commands seem to have
timeouts.
When I delete the .pyc files all is fast and happy. A few days later
things go slow and time out again.
Log when minion does not respond, doing basic test.ping:
2024-03-19 14:58:45,810 [salt.transport.zeromq:396 ][ERROR ][817]
Exception while running callback Traceback (most recent call last): File
"/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py",
line 394, in consume await callback(msg) File
"/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py",
line 484, in wrap_callback await callback(decoded) TypeError: object
NoneType can't be used in 'await' expression 2024-03-19 14:58:55,939
[salt.minion :2763][ERROR ][817] Timeout encountered while sending {'cmd':
'_return', 'id': 'cpt-ter-blahblah1.atomic.ac', 'success': True,
'return': True, 'retcode': 0, 'jid': '20240319125840713889', 'fun':
'test.ping', 'fun_args': [], 'user': 'root', '_stamp':
'2024-03-19T12:58:40.915616', 'nonce': '39f602421d8c4c9bb844bca46a7a8f83'}
request
—
Reply to this email directly, view it on GitHub
<#66177 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABRLBVGBWHGUSQGLJO25CHLYZAZX3AVCNFSM6AAAAABEKLF7COVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXGEZDMMZTHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Update, if I use --async things work ok... so something is causing minions to take very long time to respond.... but they all complete their jobs in the end. |
|
In my case , on both 3006.7 and 3007 masters, I need to restart salt-master, just to get master process works. |
|
We are also seeing |
|
Am also seeing this error with salt-minion 3007.0 on Debian 12. |
|
Anyone has issue of salt-mater not responding to simple test.ping? |
|
@dwoz , Thanks for taking on this issue. Do you need more reports to track down the root cause ? |
|
Seems to me this is basically what's happening, see lines 473-487 of client.py: >>> # This function seems to be a decorator for registering callbacks for receiving data
>>> def on_recv(callback):
... async def wrap_callback(messages):
... # decode messages
... await callback('decoded')
... return wrap_callback
...
>>> # The decorator is being used on a non-async function that returns None (instead of something awaitable)
>>> @on_recv
... def non_async_func(decoded):
... print('Non-async function called, returning None')
... return None
...
>>> # Data is received, and the callback is called
>>> import asyncio
>>> asyncio.run(non_async_func('messages'))
Non-async function called, returning None
Traceback (most recent call last):
File "<pyshell#22>", line 1, in <module>
asyncio.run(non_async_func('messages'))
File "C:\Program Files\Python312\Lib\asyncio\runners.py", line 194, in run
return runner.run(main)
File "C:\Program Files\Python312\Lib\asyncio\runners.py", line 118, in run
return self._loop.run_until_complete(task)
File "C:\Program Files\Python312\Lib\asyncio\base_events.py", line 684, in run_until_complete
return future.result()
File "<pyshell#18>", line 4, in wrap_callback
await callback('decoded')
TypeError: object NoneType can't be used in 'await' expression |
|
Doing some debugging, I don't see Would it be as simple as defining |
|
I'm not sure about others on this issue, but this issue seems to be a huge blocker. At first, we would get [No Response] and the minion log entries when running Doesn't seem like just a log entry but actually seems to be broken. |
|
@KiyoIchikawa , I have to restart salt-master to get the master time out to respond again. Running salt commands again won't fix salt-master issue. |
|
We just restarted our masters (we use the new load balancing feature) and things seem working again. We'll try changing our log level to catch any errors. |
|
See my PR #66277 that does away with the errors (so far) for me. |
|
Seeing similar issue. |
Thanks for this PR, I will try it out once my salt masters stable (without time out , not responding to salt commands issues). |
Once I enable log level to debug, in /etc/salt/master, I saw my localfs path has some link files have no destinations. I removed all those bad links.
|
|
HI Guys, just throwing my hat into this, I found that on one of my minions, what was causing it to die anytime I tried to start it up, was a config file in my /etc/salt/minion.d folder. once I removed it, the minion was stable. |
|
I am concerned this issue may not be the root cause of minion's going offline. Can someone please test the changes from #66335 that has been experiencing minion outages? @tjyang @z900collector |
|
@dwoz , AFAIK from my testing, "... await expression" in salt minion log doesn't impact salt-minion connection issue with salt-master. I opened another issue to track my salt-master issue that not able to send out commands via 4505 publish port. |
|
any news about that? this simple run tooks more than 10minutes the debug output repeats always: |
|
Please tag this as a high priority bug. |
|
same problem |
|
Even a simple mika!2006# time salt 'cal*' test.ping
calvados:
True
real 0m5.786s
user 0m0.526s
sys 0m0.071s
mika!2006# time salt 'cal*' test.ping
calvados:
True
real 0m0.639s
user 0m0.500s
sys 0m0.049s
mika!2006# time salt 'cal*' test.ping
calvados:
True
real 0m5.697s
user 0m0.529s
sys 0m0.066s
mika!2006# time salt 'cal*' test.ping
calvados:
True
real 0m0.650s
user 0m0.500s
sys 0m0.061sWhen the minion becomes unresponsive then stopping salt-minion on it takes very long and does not make a clean stop: calvados!2039# systemctl stop salt-minion
calvados!2041# pgrep -a salt
76678 /usr/bin/python /usr/bin/salt-minion
79575 /usr/bin/python /usr/bin/salt-minionSo there is obviously something hanging. calvados!2044# journalctl -fu salt-minion
Apr 29 08:15:32 calvados systemd[1]: salt-minion.service: State 'stop-sigterm' timed out. Killing.
Apr 29 08:15:32 calvados systemd[1]: salt-minion.service: Killing process 76675 (salt-minion) with signal SIGKILL.
Apr 29 08:15:32 calvados systemd[1]: salt-minion.service: Main process exited, code=killed, status=9/KILL
Apr 29 08:15:32 calvados systemd[1]: salt-minion.service: Failed with result 'timeout'.
Apr 29 08:15:32 calvados systemd[1]: salt-minion.service: Unit process 76678 (salt-minion) remains running after unit stopped.
Apr 29 08:15:32 calvados systemd[1]: salt-minion.service: Unit process 79575 (salt-minion) remains running after unit stopped.
Apr 29 08:15:32 calvados systemd[1]: Stopped The Salt Minion.
Apr 29 08:15:32 calvados systemd[1]: salt-minion.service: Consumed 6.111s CPU time, 112.6M memory peak, 0B memory swap peak.
Apr 29 08:17:43 calvados salt-minion[76678]: Minion received a SIGTERM. Exiting.
Apr 29 08:17:43 calvados salt-minion[76678]: The Salt Minion is shutdown. Minion received a SIGTERM. Exited. |
|
I just want to provide my following test.ping case.
|
|
Sorry to be pushy but is there any eta on this defect? |
|
I believe this is fixed with #66335 being merged. |
|
Things improved in 3007.1, but about 10% of minions still randomly have timeout issues. |
|
Once I found out one can bring down 3007.x STS server by following command. I downgraded salt-master to LTS version which is much more stable andn less bug |
Upgraded 3006.7 to 3007 on both salt-master and a single salt-minion
Salt minion log is recording the following errors constantly
/var/log/salt/minion
2024-03-07 15:53:49,411 [salt.transport.zeromq:396 ][ERROR ][2127705] Exception while running callback
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 394, in consume
await callback(msg)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 484, in wrap_callback
await callback(decoded)
TypeError: object NoneType can't be used in 'await' expression
/var/log/salt/master
2024-03-07 15:54:59,908 [salt.loader.lazy :531 ][ERROR ][1084696] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'
2024-03-07 15:54:59,979 [salt.loader.lazy :531 ][ERROR ][1084696] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'
2024-03-07 15:54:59,980 [salt.loader.lazy :531 ][ERROR ][1084696] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'
Initially spotted when doing a manual salt-call
looked in /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/ and spotted an old file after upgrade:
-rw-r--r-- 1 root root 9832 Mar 7 14:12 user.cpython-310.pyc
-rw-r--r-- 1 root root 440 Mar 7 14:12 value.cpython-310.pyc
-rw-r--r-- 1 root root 14016 Feb 26 09:08 vault.cpython-310.pyc
-rw-r--r-- 1 root root 15913 Mar 7 14:12 verify.cpython-310.pyc
-rw-r--r-- 1 root root 14925 Mar 7 14:12 versions.cpython-310.pyc
Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: 18.8.0
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: oracle 8.9
locale: utf-8
machine: x86_64
release: 5.15.0-101.103.2.1.el8uek.x86_64
system: Linux
version: Oracle Linux Server 8.9
The text was updated successfully, but these errors were encountered: