current cpu_affinity implementation is not happy with container technology

[grinapo]

Description

Right now cpu_affinity is a bitmask: the admin have to say which CPUs the process can me bound.

When the request cannot be fulfilled the syscall gives

2019-06-20 19:59:27,163 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1951, in cpu_affinity_set
2019-06-20 19:59:27,166 - twisted - 242 - ERROR - -     cext.proc_cpu_affinity_set(self.pid, cpus)
2019-06-20 19:59:27,167 - twisted - 242 - ERROR - - OSError: [Errno 22] Invalid argument

and homeserver panics.

This is a fine way to convey the message to the admin that the affinity flags won't work, but…

In a virtual (container) environment it is posible that a container have access to a limited set of CPUs, say, [1,2,3,4], and since the admin was suggested to pin the single-threaded python code to one CPU s/he may do a pin to cpu1.
But then the virtualisation evironment may reassign the CPUs, say, [5,7,9,11] to the container, and pin-to-1 will panic. This makes cpu_affinity basically unuseable.

Suggestion

It seems one reason to use cpu_affinity is to bound the process to one CPU and it it usually does not matter which one. It would be good if there was an alternative way to request that every thread should be affine to one (existing) cpu, no matter which. Or, alternatively, using this as a fallback to current cpu_affinity, but I don't think this added complexity would serve any good.

It seems python already handle the knowledge pretty well:

2019-06-20 19:59:27,169 - twisted - 242 - ERROR - - Traceback (most recent call last):
2019-06-20 19:59:27,170 - twisted - 242 - ERROR - -   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
2019-06-20 19:59:27,171 - twisted - 242 - ERROR - -     "__main__", mod_spec)
2019-06-20 19:59:27,171 - twisted - 242 - ERROR - -   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
2019-06-20 19:59:27,172 - twisted - 242 - ERROR - -     exec(code, run_globals)
2019-06-20 19:59:27,173 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/synapse/app/homeserver.py", line 663, in <module>
2019-06-20 19:59:27,174 - twisted - 242 - ERROR - -     main()
2019-06-20 19:59:27,174 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/synapse/app/homeserver.py", line 659, in main
2019-06-20 19:59:27,175 - twisted - 242 - ERROR - -     run(hs)
2019-06-20 19:59:27,175 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/synapse/app/homeserver.py", line 650, in run
2019-06-20 19:59:27,175 - twisted - 242 - ERROR - -     logger=logger,
2019-06-20 19:59:27,176 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/synapse/app/_base.py", line 146, in start_reactor
2019-06-20 19:59:27,176 - twisted - 242 - ERROR - -     run()
2019-06-20 19:59:27,176 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/synapse/app/_base.py", line 125, in run
2019-06-20 19:59:27,176 - twisted - 242 - ERROR - -     p.cpu_affinity(cpus)
2019-06-20 19:59:27,177 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 933, in cpu_affinity
2019-06-20 19:59:27,177 - twisted - 242 - ERROR - -     self._proc.cpu_affinity_set(list(set(cpus)))
2019-06-20 19:59:27,177 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1514, in wrapper
2019-06-20 19:59:27,178 - twisted - 242 - ERROR - -     return fun(self, *args, **kwargs)
2019-06-20 19:59:27,178 - twisted - 242 - ERROR - -   File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1964, in cpu_affinity_set
2019-06-20 19:59:27,179 - twisted - 242 - ERROR - -     "between %s" % (cpu, eligible_cpus))
2019-06-20 19:59:27,179 - twisted - 242 - ERROR - - ValueError: CPU number 0 is not eligible; choose between [2, 3, 4]

Version information

v1.0.0, platform linux, lxc based containers.

[richvdh]

as per #5525: cpu_affinity isn't much use on python3, and we're removing support for it.

I think restricting synapse to one CPU (and only one CPU) is somewhat out of scope for the application-level configuration, and is better managed with OS-level tools such as cgroups. (It's worth noting that synapse is largely single-threaded anyway, so I'm not sure that applying such a limit will gain you much.)