io: improve bytes handling

[sobolevn]

IncrementalNewlineDecoder.decode does not support str, but supports bytearray and memoryview:

>>> import codecs, io
>>> dec = codecs.getincrementaldecoder('utf8')()
>>> d = io.IncrementalNewlineDecoder(dec, translate=True)
>>> d.decode('str')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sobolev/.pyenv/versions/3.10.0/lib/python3.10/codecs.py", line 321, in decode
    data = self.buffer + input
TypeError: can't concat str to bytes

Buffers:

>>> d.decode(bytearray(b'\r'))
'\n'
>>> d.decode(memoryview(b'\r'))
'\n'
>>> d.decode(b'\r')
'\n'

[srittau]

It isn't that easy, unfortunately:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import io
>>> d = io.IncrementalNewlineDecoder(None, translate=True)
>>> d.decode('str')
'str'
>>> d.decode(b'bytes')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: decoder should return a string result, not 'bytes'

This is the implemenation: https://github.com/python/cpython/blob/934b25dcc492dcbca4da9d63d0d71dc940fc0375/Modules/_io/textio.c#L284

PyObject *
_PyIncrementalNewlineDecoder_decode(PyObject *myself,
                                    PyObject *input, int final)
{
    PyObject *output;
    Py_ssize_t output_len;
    nldecoder_object *self = (nldecoder_object *) myself;

    if (self->decoder == NULL) {
        PyErr_SetString(PyExc_ValueError,
                        "IncrementalNewlineDecoder.__init__ not called");
        return NULL;
    }

    /* decode input (with the eventual \r from a previous pass) */
    if (self->decoder != Py_None) {
        output = PyObject_CallMethodObjArgs(self->decoder,
            &_Py_ID(decode), input, final ? Py_True : Py_False, NULL);
    }
    else {
        output = input;
        Py_INCREF(output);
    }

    if (check_decoded(output) < 0)
        return NULL;

    /* ... */

So, there are code paths, where only str is allowed, as check_decoded() explicity checks for a str.

[sobolevn]

Ok, let's go with an easy solution. I don't have any other ideas :(

[JelleZijlstra]

I'm not sure this is quite right either. The argument gets passed to codecs.IncrementalDecoder.decode, which is an abstract method. I left it as taking bytes when I reviewed codecs, because that's what BufferedIncrementalDecoder initializes its buffer to. There may be decoders that accept ReadableBuffer, but I'm not sure it's required. If we make the change in this PR, we should also change codecs.

[sobolevn]

Yes, you are correct. Here's the full sequence:

(abstract)codecs.IncrementalDecoder -> (abstract)codecs.BufferedIncrementalDecoder -> encodings.*.IncrementalDecoder

Then io.IncrementalNewlineDecoder uses some of the concrete types from encodings.*.
Types in encodings.* already have ReadableBuffer set in this PR: #9043

So, I will create a new PR to fix codecs. types in a moment.

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[sobolevn]

There's an unrelated failure in the CI.