Wrong line numbers reported under Python 3.8 with v0.2.x

[shifqu]

Hello

Given the following code snippet

import json


def another(x):
    """Another test function.

    This function is here to add extra lines to this snippet

    Parameters
    ----------
    x : int
        An arbitrary variable
    """
    json.loads(str(x))


# This is line 17
def f(id_):
    """Test function.

    Parameters
    ----------
    id_ : str. # This is line 23
        The identifier of the object
    """
    print("Hello world!")

I have the following flake8 configuration (nothing specific for rst-docstrings, but including for completeness' sake):

[flake8]
max_line_length = 120
max_complexity = 10
doctests = True
ignore = S101,E501,E203,W503,FS003

Python version:
Python 3.8.8

Then when I run the flake8 (with flake8-rst-docstrings==0.2.1) on the following snippet I receive:
/SCRATCHES/poc_flake8_docstrings.py:17:1: RST306 Unknown target name: "id".

when I run the flake8 (with flake8-rst-docstrings==0.1.2) on the following snippet I receive:
/SCRATCHES/poc_flake8_docstrings.py:23:1: RST306 Unknown target name: "id".

As you can observe, when using the latest version (happens for 0.2.0 as well), flake8 shows us the wrong line number.
Since I don't wan't to fix the doc, but ignore it. I now have to place a # noqa: RST306 on line 17. But in my real code, I have my function decorated, so I have to put the noqa behind an argument in my decorator and that makes no sense at all.

The behaviour is actually wrong as well, but this is mentioned in #26

If you need any more info, or if I can be of any more assistance, don't hesitate :)

[peterjc]

All the line numbers that I'd checked seemed OK, so a test case is great.

[peterjc]

Hmm. It is working for me:

$ flake8 bug.py
bug.py:1:1: D100 Missing docstring in public module
bug.py:23:1: RST306 Unknown target name: "id".

So, are we using the same things?

$ flake8 --version
3.8.1 (assertive: 1.2.1, flake8-blind-except: 0.1.1, flake8-bugbear: 20.11.1,
flake8-comprehensions: 3.2.2, flake8-docstrings: 1.5.0, pydocstyle: 5.0.2,
flake8-sfs: 0.0.3, flake8_implicit_str_concat: 0.1.0, mccabe: 0.6.1,
pycodestyle: 2.6.0, pyflakes: 2.2.0, rst-docstrings: 0.2.1) CPython 3.7.8 on
Darwin

This might be something in Python and/or flake8 from the AST parsing side...

[shifqu]

My output for flake8 --version

❯ flake8 --version
3.9.1 () CPython 3.8.8 on Darwin

I notice that your output is much less concise.

[peterjc]

There is something broken with your flake8 version information, it should be listing all the registered plugins and their versions. However, you do have a newer flake8, and a newer Python.

Testing on a Linux system I have access to, works fine but again Python 3.7.x and flake8 v3.8.x

$ flake8 bug.py 
bug.py:23:1: RST306 Unknown target name: "id".

$ flake8 --version
3.8.4 (flake8-blind-except: 0.2.0, flake8-bugbear: 20.11.1, mccabe: 0.6.1,
pycodestyle: 2.6.0, pyflakes: 2.2.0, rst-docstrings: 0.2.1) CPython 3.7.9 on
Linux

I updated flake8 here,

$ flake8 --version
3.9.1 (flake8-blind-except: 0.2.0, flake8-bugbear: 20.11.1, mccabe: 0.6.1,
pycodestyle: 2.7.0, pyflakes: 2.3.1, rst-docstrings: 0.2.1) CPython 3.7.9 on
Linux
$ flake8 bug.py 
bug.py:23:1: RST306 Unknown target name: "id".

No change:

Looks like time to try Python 3.8.x and/or flake8 3.9.x on the Mac to match yours.

[peterjc]

OK, so this is not due to flake8 version 3.9.1 on Mac,

$ flake8 --version
3.9.1 (assertive: 1.2.1, flake8-blind-except: 0.1.1, flake8-bugbear: 20.11.1,
flake8-comprehensions: 3.2.2, flake8-docstrings: 1.5.0, pydocstyle: 5.0.2,
flake8-sfs: 0.0.3, flake8_implicit_str_concat: 0.1.0, mccabe: 0.6.1,
pycodestyle: 2.7.0, pyflakes: 2.3.1, rst-docstrings: 0.2.1) CPython 3.7.8 on
Darwin
$ flake8 bug.py 
bug.py:1:1: D100 Missing docstring in public module
bug.py:23:1: RST306 Unknown target name: "id".

It might be due to the Python version, given AST parsing is involved.

[peterjc]

Yes, something changed with Python 3.8 - I can see this on TravisCI too, e.g.

https://travis-ci.org/github/peterjc/flake8-rst-docstrings/jobs/768133481 Python 3.4 and
https://travis-ci.org/github/peterjc/flake8-rst-docstrings/jobs/768133482 Python 3.5 and
https://travis-ci.org/github/peterjc/flake8-rst-docstrings/jobs/768133483 Python 3.6 and
https://travis-ci.org/github/peterjc/flake8-rst-docstrings/jobs/768133484 Python 3.7 include:

$ flake8 --select RST RST307/code_invalid_arg.py
RST307/code_invalid_arg.py:14:1: RST307 Error in "code" directive:
RST307/code_invalid_arg.py:23:1: RST307 Error in "code-block" directive:
RST307/code_invalid_arg.py:31:1: RST307 Error in "code-block" directive:
Good, RST307 violations reported, as expected.

https://travis-ci.org/github/peterjc/flake8-rst-docstrings/jobs/768133485 Python 3.8 includes:

$ flake8 --select RST RST307/code_invalid_arg.py
RST307/code_invalid_arg.py:-27:1: RST307 Error in "code" directive:
RST307/code_invalid_arg.py:-18:1: RST307 Error in "code-block" directive:
RST307/code_invalid_arg.py:-10:1: RST307 Error in "code-block" directive:
Good, RST307 violations reported, as expected.

[peterjc]

It seems https://bugs.python.org/issue33416 explicitly added .end_lineno and .end_col_offset attributes for Python 3.8, but have not found anything about fixing the .lineno value (I was seeing giving the end line of the docstring but I wanted the start line).

There seems to have been related changes in Python 3.7, e.g. https://bugs.python.org/issue30497 and PyCQA/pyflakes#271

I have a possible fix in hand...

[shifqu]

Thanks a lot for your prompt response, @peterjc.
Eagerly awaiting the fix you have in mind 👀

[peterjc]

The fix is working, just struggling with some off by one "fun" between the different versions.

[peterjc]

@shifqu are you in a position to test #37 now (in the next few hours), or should I go ahead and do a point release?

[shifqu]

Thanks. I tested it locally on my MacOS and the line numbers are correctly reported again. 🎉

❯ flake8 --version
3.9.1 (flake-mutable: 1.2.0, flake8-bandit: 2.1.2, flake8-bugbear: 21.4.3, flake8-comprehensions: 3.4.0, flake8-pytest-style: 1.4.1, mccabe: 0.6.1, pycodestyle:
2.7.0, pyflakes: 2.3.1, **_rst-docstrings: 0.2.2_**, use-fstring-format: 1.0, use-fstring-percent: 1.0, use-fstring-prefix: 1.0) CPython 3.8.8 on Darwin

[peterjc]

That's v0.2.2 released on PyPI now, conda-forge will follow usually within a day.

Thank you for taking the time to report this, and help test my changes.

[shifqu]

Great! Thank you for the prompt response and resolution.