Frequent Timeout Errors in Marzban Logs

[sahmadiut]

Frequent Timeout Errors in Marzban Logs

Description

After upgrading the Xray core version from 1.8.24 to 24.12.31, we started encountering frequent timeout errors in the Marzban logs. This issue occurs every few hours, despite the nodes and master being fully connected and operational. Both the nodes and master are using the same version of the Xray core.

Error Details

Below is an excerpt from the logs:

marzban-marzban-1  | Job "core_health_check (trigger: interval[0:00:10], next run at: 2025-01-16 11:35:53 UTC)" raised an exception
marzban-marzban-1  | Traceback (most recent call last):
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
marzban-marzban-1  |     six.raise_from(e, None)
marzban-marzban-1  |   File "<string>", line 3, in raise_from
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 462, in _make_request
marzban-marzban-1  |     httplib_response = conn.getresponse()
marzban-marzban-1  |                        ^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/http/client.py", line 1428, in getresponse
marzban-marzban-1  |     response.begin()
marzban-marzban-1  |   File "/usr/local/lib/python3.12/http/client.py", line 331, in begin
marzban-marzban-1  |     version, status, reason = self._read_status()
marzban-marzban-1  |                               ^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/http/client.py", line 292, in _read_status
marzban-marzban-1  |     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
marzban-marzban-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/socket.py", line 720, in readinto
marzban-marzban-1  |     return self._sock.recv_into(b)
marzban-marzban-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/ssl.py", line 1251, in recv_into
marzban-marzban-1  |     return self.read(nbytes, buffer)
marzban-marzban-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/ssl.py", line 1103, in read
marzban-marzban-1  |     return self._sslobj.read(len, buffer)
marzban-marzban-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  | TimeoutError: The read operation timed out
marzban-marzban-1  | 
marzban-marzban-1  | During handling of the above exception, another exception occurred:
marzban-marzban-1  | 
marzban-marzban-1  | Traceback (most recent call last):
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
marzban-marzban-1  |     resp = conn.urlopen(
marzban-marzban-1  |            ^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 801, in urlopen
marzban-marzban-1  |     retries = retries.increment(
marzban-marzban-1  |               ^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 552, in increment
marzban-marzban-1  |     raise six.reraise(type(error), error, _stacktrace)
marzban-marzban-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/urllib3/packages/six.py", line 770, in reraise
marzban-marzban-1  |     raise value
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
marzban-marzban-1  |     httplib_response = self._make_request(
marzban-marzban-1  |                        ^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 469, in _make_request
marzban-marzban-1  |     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
marzban-marzban-1  |     raise ReadTimeoutError(
marzban-marzban-1  | urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='<< node address >>', port=62050): Read timed out. (read timeout=3)
marzban-marzban-1  | 
marzban-marzban-1  | During handling of the above exception, another exception occurred:
marzban-marzban-1  | 
marzban-marzban-1  | Traceback (most recent call last):
marzban-marzban-1  |   File "/code/app/xray/node.py", line 104, in make_request
marzban-marzban-1  |     res = self.session.post(self._rest_api_url + path, timeout=timeout,
marzban-marzban-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 637, in post
marzban-marzban-1  |     return self.request("POST", url, data=data, json=json, **kwargs)
marzban-marzban-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
marzban-marzban-1  |     resp = self.send(prep, **send_kwargs)
marzban-marzban-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
marzban-marzban-1  |     r = adapter.send(request, **kwargs)
marzban-marzban-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
marzban-marzban-1  |     raise ReadTimeout(e, request=request)
marzban-marzban-1  | requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='<< node address >>', port=62050): Read timed out. (read timeout=3)
marzban-marzban-1  | 
marzban-marzban-1  | During handling of the above exception, another exception occurred:
marzban-marzban-1  | 
marzban-marzban-1  | Traceback (most recent call last):
marzban-marzban-1  |   File "/usr/local/lib/python3.12/site-packages/apscheduler/executors/base.py", line 125, in run_job
marzban-marzban-1  |     retval = job.func(*job.args, **job.kwargs)
marzban-marzban-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/code/app/jobs/0_xray_core.py", line 24, in core_health_check
marzban-marzban-1  |     assert node.started
marzban-marzban-1  |            ^^^^^^^^^^^^
marzban-marzban-1  |   File "/code/app/xray/node.py", line 129, in started
marzban-marzban-1  |     res = self.make_request("/", timeout=3)
marzban-marzban-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
marzban-marzban-1  |   File "/code/app/xray/node.py", line 109, in make_request
marzban-marzban-1  |     raise exc
marzban-marzban-1  | app.xray.node.NodeAPIError: (0, "HTTPSConnectionPool(host='<< node address >>', port=62050): Read timed out. (read timeout=3)")

Steps to Reproduce

1. Upgrade Xray core from version 1.8.24 to 24.12.31.
2. Ensure both master and nodes are running the same version.
3. Observe logs over time. The error surfaces intermittently during the core_health_check.

Expected Behavior

The core_health_check job should execute without throwing timeout errors, maintaining consistent communication with the nodes.

Actual Behavior

The health check job intermittently fails with a timeout error, as described in the logs.

Additional Context

• Frequency: The issue appears roughly every few hours.
• Environment:
  • Marzban version: v0.8.4
  • Python version: 3.12
  • Xray core version: 24.12.31
• There are no connectivity issues between the master and nodes when tested manually.

Suggestions for Debugging

• Investigate if any changes in the Xray core version 24.12.31 affect API response times.
• Check timeout settings for the core_health_check job and evaluate if they need adjustment.
• Verify if this issue is reproducible with earlier Xray core versions.