ResourceWarning: unclosed <socket.socket> #1293
Comments
|
Still a problem in version 2.4.2 (observed on python:3.6.1-alpine) |
|
Have you tried calling |
|
Even with that, I still get the occasional ResourceWarning error. Why doesn't Minimal reproducer: #!/usr/bin/env python3
import docker
import logging
logging.basicConfig(level=logging.DEBUG)
d = docker.from_env()
c = d.containers.run('alpine', auto_remove=True, detach=True, command='sleep 10')
res = c.exec_run('echo foo')
print(res)
c.kill()
d.close()Debug patch: --- a/usr/local/lib/python3.7/site-packages/docker/transport/unixconn.py
+++ b/usr/local/lib/python3.7/site-packages/docker/transport/unixconn.py
@@ -42,6 +42,15 @@
sock.settimeout(self.timeout)
sock.connect(self.unix_socket)
self.sock = sock
+ print("Connected: %r" % self.sock)
+ if sock.fileno() in [6, 7]:
+ import traceback; traceback.print_stack()
+
+ def close(self):
+ print("closing: %r" % self.sock)
+ if self.sock.fileno() == 6:
+ import traceback; traceback.print_stack()
+ super().close()
def putheader(self, header, *values):
super(UnixHTTPConnection, self).putheader(header, *values)Tested with Python 3.7.3 and docker-py 3.7.2 using: Command output |
Lekensteyn
added a commit
to Lekensteyn/docker-py
that referenced
this issue
Apr 26, 2019
Requests with stream=True MUST be closed or else the connection will
never be returned to the connection pool. Both ContainerApiMixin.attach
and ExecApiMixin.exec_start were leaking in the stream=False case.
exec_start was modified to follow attach for the stream=True case as
that allows the caller to close the stream when done (untested).
Tested with:
# Test exec_run (stream=False) - observe one less leak
make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning'
# Test exec_start (stream=True, fully reads from CancellableStream)
make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning'
After this change, one resource leak is removed, the remaining resource
leaks occur because none of the tests call client.close().
Fixes docker#1293
(Regression from docker#1130)
Signed-off-by: Peter Wu <[email protected]>
Lekensteyn
added a commit
to Lekensteyn/docker-py
that referenced
this issue
Apr 26, 2019
Requests with stream=True MUST be closed or else the connection will
never be returned to the connection pool. Both ContainerApiMixin.attach
and ExecApiMixin.exec_start were leaking in the stream=False case.
exec_start was modified to follow attach for the stream=True case as
that allows the caller to close the stream when done (untested).
Tested with:
# Test exec_run (stream=False) - observe one less leak
make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning'
# Test exec_start (stream=True, fully reads from CancellableStream)
make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning'
After this change, one resource leak is removed, the remaining resource
leaks occur because none of the tests call client.close().
Fixes docker#1293
(Regression from docker#1130)
Signed-off-by: Peter Wu <[email protected]>
devplayer0
added a commit
to devplayer0/docker-net-dhcp
that referenced
this issue
Sep 22, 2019
The file descriptor problem is multi-tiered... Maintaining pyroute2 NDB sources in namespaces keeps a proxy process running in each namespace, wasting a lot of file descriptors on pipes. It also leaks some of these pipes upon removal of sources! Even the Python Docker client leaks its sockets! (docker/docker-py#1293)
|
Has this been fixed yet? I get this with docker py v4.2.0 |
|
@caaespin presumably not, https://github.com/docker/docker-py/blob/master/docker/api/client.py has not been modified since March 15, 2019 while PR #2320 is still open. |
riyad
pushed a commit
to riyad/docker-py
that referenced
this issue
Jan 7, 2022
Requests with stream=True MUST be closed or else the connection will
never be returned to the connection pool. Both ContainerApiMixin.attach
and ExecApiMixin.exec_start were leaking in the stream=False case.
exec_start was modified to follow attach for the stream=True case as
that allows the caller to close the stream when done (untested).
Tested with:
# Test exec_run (stream=False) - observe one less leak
make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning'
# Test exec_start (stream=True, fully reads from CancellableStream)
make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning'
After this change, one resource leak is removed, the remaining resource
leaks occur because none of the tests call client.close().
Fixes docker#1293
(Regression from docker#1130)
Signed-off-by: Peter Wu <[email protected]>
|
as a workaround: def docker_from_env() -> ContextManager[docker.DockerClient]:
return contextlib.closing(docker.from_env())usage: with docker_from_env() as client:
... |
asottile-sentry
added a commit
to getsentry/sentry
that referenced
this issue
Sep 15, 2022
diff is best viewed with [?w=1](https://github.com/getsentry/sentry/pull/38906/files?w=1) working around docker/docker-py#1293 before: ```console $ pytest -Wonce tests/symbolicator/test_minidump_full.py::SymbolicatorMinidumpIntegrationTest::test_full_minidump ============================= test session starts ============================== platform darwin -- Python 3.8.13, pytest-7.1.2, pluggy-0.13.1 rootdir: /Users/asottile/workspace/sentry, configfile: pyproject.toml plugins: forked-1.4.0, rerunfailures-10.2, xdist-2.4.0, sentry-0.1.9, django-4.4.0, cov-3.0.0 collected 1 item tests/symbolicator/test_minidump_full.py . [100%] =============================== warnings summary =============================== tests/symbolicator/test_minidump_full.py::SymbolicatorMinidumpIntegrationTest::test_full_minidump /Users/asottile/workspace/sentry/.venv/lib/python3.8/site-packages/django/utils/regex_helper.py:147: ResourceWarning: unclosed <socket.socket fd=17, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/asottile/Library/Containers/com.docker.docker/Data/docker.raw.sock> result.append(Group((("%%(%s)s" % param), param))) Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. tests/symbolicator/test_minidump_full.py::SymbolicatorMinidumpIntegrationTest::test_full_minidump /Users/asottile/workspace/sentry/.venv/lib/python3.8/site-packages/django/utils/regex_helper.py:147: ResourceWarning: unclosed <socket.socket fd=18, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/asottile/Library/Containers/com.docker.docker/Data/docker.raw.sock> result.append(Group((("%%(%s)s" % param), param))) Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. tests/symbolicator/test_minidump_full.py::SymbolicatorMinidumpIntegrationTest::test_full_minidump /Users/asottile/workspace/sentry/.venv/lib/python3.8/site-packages/django/utils/regex_helper.py:147: ResourceWarning: unclosed <socket.socket fd=19, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/asottile/Library/Containers/com.docker.docker/Data/docker.raw.sock> result.append(Group((("%%(%s)s" % param), param))) Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================== 1 passed, 3 warnings in 12.34s ======================== %4|1663268253.516|TERMINATE|rdkafka#producer-4| [thrd:app]: Producer terminating with 3 messages (539 bytes) still in queue or transit: use flush() to wait for outstanding message delivery ``` after: ```console $ pytest -Wonce tests/symbolicator/test_minidump_full.py::SymbolicatorMinidumpIntegrationTest::test_full_minidump ============================= test session starts ============================== platform darwin -- Python 3.8.13, pytest-7.1.2, pluggy-0.13.1 rootdir: /Users/asottile/workspace/sentry, configfile: pyproject.toml plugins: forked-1.4.0, rerunfailures-10.2, xdist-2.4.0, sentry-0.1.9, django-4.4.0, cov-3.0.0 collected 1 item tests/symbolicator/test_minidump_full.py . [100%] ============================== 1 passed in 12.33s ============================== %4|1663268283.383|TERMINATE|rdkafka#producer-4| [thrd:app]: Producer terminating with 3 messages (539 bytes) still in queue or transit: use flush() to wait for outstanding message delivery ```
gaelL
added a commit
to cycloidio/docker-cycloid-toolkit
that referenced
this issue
Dec 14, 2022
assertEquals -> assertEqual And fix docker socket unclosed warning eg docker/docker-py#1293
milas
added a commit
that referenced
this issue
Jan 13, 2023
Requests with stream=True MUST be closed or else the connection will
never be returned to the connection pool. Both ContainerApiMixin.attach
and ExecApiMixin.exec_start were leaking in the stream=False case.
exec_start was modified to follow attach for the stream=True case as
that allows the caller to close the stream when done (untested).
Tested with:
# Test exec_run (stream=False) - observe one less leak
make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning'
# Test exec_start (stream=True, fully reads from CancellableStream)
make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning'
After this change, one resource leak is removed, the remaining resource
leaks occur because none of the tests call client.close().
Fixes #1293
(Regression from #1130)
Signed-off-by: Peter Wu <[email protected]>
Co-authored-by: Milas Bowman <[email protected]>
felixfontein
added a commit
to felixfontein/community.docker
that referenced
this issue
Feb 11, 2023
…py#2320) Requests with stream=True MUST be closed or else the connection will never be returned to the connection pool. Both ContainerApiMixin.attach and ExecApiMixin.exec_start were leaking in the stream=False case. exec_start was modified to follow attach for the stream=True case as that allows the caller to close the stream when done (untested). Tested with: # Test exec_run (stream=False) - observe one less leak make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning' # Test exec_start (stream=True, fully reads from CancellableStream) make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning' After this change, one resource leak is removed, the remaining resource leaks occur because none of the tests call client.close(). Fixes docker/docker-py#1293 (Regression from docker/docker-py#1130) Cherry-picked from docker/docker-py@34e6829 Co-authored-by: Peter Wu <[email protected]> Co-authored-by: Milas Bowman <[email protected]>
felixfontein
added a commit
to ansible-collections/community.docker
that referenced
this issue
Feb 12, 2023
…py#2320) (#582) Requests with stream=True MUST be closed or else the connection will never be returned to the connection pool. Both ContainerApiMixin.attach and ExecApiMixin.exec_start were leaking in the stream=False case. exec_start was modified to follow attach for the stream=True case as that allows the caller to close the stream when done (untested). Tested with: # Test exec_run (stream=False) - observe one less leak make integration-test-py3 file=models_containers_test.py' -k test_exec_run_success -vs -W error::ResourceWarning' # Test exec_start (stream=True, fully reads from CancellableStream) make integration-test-py3 file=api_exec_test.py' -k test_execute_command -vs -W error::ResourceWarning' After this change, one resource leak is removed, the remaining resource leaks occur because none of the tests call client.close(). Fixes docker/docker-py#1293 (Regression from docker/docker-py#1130) Cherry-picked from docker/docker-py@34e6829 Co-authored-by: Peter Wu <[email protected]> Co-authored-by: Milas Bowman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello.
It seems that in certain situations http socket isn't properly closed.
Python 3.5.1 on OSX.
Can be reproduced with following script:
The text was updated successfully, but these errors were encountered: