'NoneType' object has no attribute 'resume_reading'

[thehesiod]

Checklist

• The bug is reproducible against the latest release and/or master.
• There are no similar issues or pull requests to fix it yet.

Describe the bug

I recently tried switching to httpx because of instability with aiohttp, however after a few tens of thousands of requests against URLs like https://hydro1.gesdisc.eosdis.nasa.gov/data/NLDAS/NLDAS_FORA0125_H.002/2020/172/NLDAS_FORA0125_H.A20200620.1000.002.grb with query string: {'If-Modified-Since': 'Mon, 17 Aug 2009 16:20:16 GMT'} with 10 workers I got an error stack like this:

Traceback (most recent call last):
  File "/usr/local/fbn/fbn.com/api/weather/python/fbn/asyncio/data_transfer.py", line 170, in cacheable_get
    r = await self._client.get(url, headers=headers, params=params)
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpx/_client.py", line 1305, in get
    return await self.request(
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpx/_client.py", line 1147, in request
    response = await self.send(
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpx/_client.py", line 1174, in send
    await response.aread()
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpx/_models.py", line 965, in aread
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpx/_models.py", line 965, in <listcomp>
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpx/_models.py", line 976, in aiter_bytes
    async for chunk in self.aiter_raw():
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpx/_models.py", line 1009, in aiter_raw
    async for part in self._raw_stream:
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 49, in __aiter__
    async for chunk in self.stream:
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpcore/_async/base.py", line 57, in __aiter__
    async for chunk in self.aiterator:
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpcore/_async/http11.py", line 128, in _receive_response_data
    event = await self._receive_event(timeout)
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpcore/_async/http11.py", line 145, in _receive_event
    data = await self.socket.read(self.READ_NUM_BYTES, timeout)
  File "/root/.local/share/virtualenvs/weather-h8QMrSBh/lib/python3.8/site-packages/httpcore/_backends/asyncio.py", line 134, in read
    return await asyncio.wait_for(
  File "/usr/local/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/usr/local/lib/python3.8/asyncio/streams.py", line 690, in read
    self._maybe_resume_transport()
  File "/usr/local/lib/python3.8/asyncio/streams.py", line 461, in _maybe_resume_transport
    self._transport.resume_reading()
  File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 344, in resume_reading
    self._ssl_protocol._transport.resume_reading()
AttributeError: 'NoneType' object has no attribute 'resume_reading'

It seems similar to what's reported here: https://bugs.python.org/issue36098. so may be an underlying asyncio bug.

To reproduce

Reproduction steps are like above, it takes many, many many requests until it happens. Please let me know how I can help, however I think there's enough information here that a testcase shouldn't be needed.

Expected behavior

Should never trigger an attribute error

Actual behavior

Attribute error happens after 10s of thousands of requests

Debugging material

see above

Environment

• OS: Debian Buster
• Python version: 3.8.2
• HTTPX version: 0.13.3
• Async environment: asyncio
• HTTP proxy: no
• Custom certificates: no

Additional context

[thehesiod]

with this error I have no confidence the pool hasn't been polluted, can someone comment as to how bad this is?

[florimondmanca]

Hi @thehesiod

In general what helps a lot in this kind of issue is a stand-alone reproduction snippet, with a corresponding well-known server setup. I see the nasa.gov endpoint you're targeting is (obviously) protected behind auth, so I can't reproduce, and besides we don't know what kind of server setup that endpoint is running (the bpo ticket mentions possibly "slow SSL" but I'm not sure what that means in practice).

Are you able to reproduce the bug by running the reproduction snippet in https://bugs.python.org/issue36098, stripping the server code and running the client with HOST/PORT pointing to your nasa.gov target? If so, this sounds like an "external bug" candidate, unfortunately. Someone on the bpo ticket mentioned uvloop (alternative asyncio event loop impl) might not have this problem, so you could try that out.

[thehesiod]

access is actually free: https://disc.gsfc.nasa.gov/data-access. However I was hoping you guys would have a work-around as libraries should try avoiding breaking catastrophically :). ex: internally catching this, throwing away the connector and raising some sort of broken pool exception or something. I'm pretty sure it's triggered by this underlying asyncio issue

[yeraydiazdiaz]

libraries should try avoiding breaking catastrophically :)

Please be aware of your tone.

You're admitting it's an underlying asyncio issue. Your use case is very specific and triggered after tens of thousands of requests, yet you're passive-aggressively accusing the authors of not forseeing it or having a work around for it.

[thehesiod]

I'm not sure where you're inferring all that. I'm a friendly guy and just asking if this library can handle an issue it could not have foreseen or its fault. I'm not saying there's anything wrong with httpx, the issue is most likely the underlying asyncio module and there's no way this library could have foreseen that. Here's all what I'm asking:

Is there a way for httpx to be able to handle this asyncio issue so that it will be in a "good" state if asyncio throws a grenade at us (pool still valid and bad connector dropped)...maybe this is already true however as a client I have no way to know this since an assert is throw from the bowels of asyncio. This is why I was implying that it would be nice if this were caught and handled. As a client I can't use this library without some sort of behavior like that as I won't know if the error is coming from httpx or some other library and if it's ok to ignore.

This is a nice to have basically, would allow me to use this library until an asyncio fix comes out (if ever).

Any use case can be considered "very specific". I don't think that's a way to approach any issue.

[florimondmanca]

@thehesiod I'm joining Yeray's comment here: reading your comments I'm under the impression that you're expecting us to fix things for you. I don't think that's what you're meaning to convey, but any case, that's now how I envision open source to work.

To answer your question - I personally cannot guarantee that just catching the AttributeError on your side (which can be done fairly safely by matching the error message) wouldn't let the connection dangling around, but I'm not sure that it would be causing a problem either. So you can try this as a workaround, and see if anything goes wrong (e.g. HTTPX trying to reuse a connection without knowing the underlying socket is broken). If this doesn't solve this for you, then you can consider investigating further what exactly would need to change in HTTPX to address your use case, and submit a pull request.

In the meantime I'm going to close this as an "external bug" in asyncio. Thanks all!

[thehesiod]

reproduced here: https://repl.it/repls/PristineBonyBugs#main.py. what's funny is it doesn't reproduce locally, but very easily on this site

[thehesiod]

it just uses httpx + core asyncio server

[thehesiod]

another datapoint, this does not seem to reproduce with aiohttp: https://repl.it/@thehesiod/PristineBonyBugs-1#main.py. so there's something specific about how httpx interacts with the underlying asyncio subsystem

[thehesiod]

updated testcase to not require gather and now it happens immediately on that site and locally

[thehesiod]

so investigating this some, it seems like httpx is doing an extra read after the connection is closed, whereas aiohttp does not. So this may in fact be an httpx bug. Ideally the _SSLProtocolTransport.resume_reading method would be a no-op if the connection is already closed, however I'm not sure what assumptions its making. But the question remains why is httpx doing an extra read as compared to aiohttp

[thehesiod]

So boiling this down I think there are two actionables:

1. Why does httpx not work while aiohttp does (needs httpx protocol level investigation)
2. I'm going to test patching the resume_reading method to see if it works ok by adding a null check. I tested just catching and retrying the exception but it didn't seem to work.

I'm afraid of 2) potentially causing data corruption. In my particular case I think I can do some ETag validation. I'll report back if two doesn't work. But need helper with 1)