Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-29428: make doctest documentation clearer #45

Closed
wants to merge 3 commits into from
Closed

bpo-29428: make doctest documentation clearer #45

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
158219b
bpo-29428: make doctest documentation clearer
Feb 12, 2017
a140a7f
Address bitdancer review comments, Doc/library/doctest.rst
Feb 13, 2017
62a9a46
Fix typo "an test" -> "a test"
Feb 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 53 additions & 27 deletions Doc/library/doctest.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,8 +276,9 @@ sections.
Which Docstrings Are Examined?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The module docstring, and all function, class and method docstrings are
searched. Objects imported into the module are not searched.
The docstring for the module, and the docstrings for all functions,
classes, and methods in that module, are searched.
Objects imported into the module are not searched.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the existing wording here. Your formulation uses more words without adding any clarity. This isn't a strong preference, however, so if other people prefer it I have no object. However, 'for the' should be 'of', since the docstring belongs to the object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, thank you, David, for your careful review. You have found many ways to improve this PR.

The phrase "The module docstring" caused me to stumble as I read it. The first meaning that came to mind was "The module known as docstring", that is, a module. Of course the intended meaning is "The docstring of the module", that is, a docstring. I think the original wording affords two interpretations, and the replacement affords only one. But, I don't feel strongly about this change. I just improved it on my way to the next section.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for the @JDLH wording.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for disambiguating, can probably be made short again using something like:

The docstring of the module, and all function, class and method docstrings are searched.


In addition, if ``M.__test__`` exists and "is true", it must be a dict, and each
entry maps a (string) name to a function object, class object, or string.
Expand All @@ -300,36 +301,54 @@ their contained methods and nested classes.
How are Docstring Examples Recognized?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

A doctest example is composed of one or more tests. An individual test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks wrong, an example (that you name test) starts with >>> and ends after its output.

Meaning this is a correct example not covered by your paragraph:

>>> say = "Ni"
>>> print(say)
Ni

This is already covered by:

Any expected output must immediately follow the final '>>> ' or '... ' line containing the code, and the expected output (if any) extends to the next '>>> ' or all-whitespace line.

Please remove this paragraph.

starts with a line that starts with '>>>', has zero or more code
continuation lines that start with '...', and ends with zero or more
expected output lines. The expected output ends at the first line that
starts with '>>>' or is blank. All lines in an example block must have
the same indentation level.

In most cases a copy-and-paste of an interactive console session works fine,
but doctest isn't trying to do an exact emulation of any specific Python shell.
but doctest isn't trying to do an exact emulation of the Python shell.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There exist more than one Python shell, please leave "any specific".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reviewers are not consistent. Back on the first commit, I kept the wording "any specific Python shell". @bitdancer 's comment on Feb 12, 2017 was:

Perhaps this should say "the python shell" instead of "any specific python shell", since I think most alternate shells can't be cut and pasted as doctests.
My reply was:
This sentence is verbatim from the previous version of the document. I don't have strong feelings about this change. I'm happy to go with "the python shell".

What wording will satisfy both @bitdancer and @JulienPalard ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the inconsistencies, both are true, I still prefer "any specific", but "the" is simplier. In any cases the message passes about "don't rely on copy/paste to work 100% of the times".

Again, this modification is not linked to your original issue, when we're saying "try to propose atomic changes" we're having this in mind: The more you're modifying, the more people will discuss it, this is also why I originally said "please leave" this as is.


::

>>> # comments are ignored
>>> x = 12
>>> x
12
>>> if x == 13:
... print("yes")
>>> import math
>>> x = factorial(10); math.ceil(math.log10(x))
7
>>> if x == factorial(9)*10:
... print("same:\n{0}".format(x))
... else:
... print("no")
... print("NO")
... print("NO!!!")
... print("differ:\n{0}\n{1}".format(x, factorial(9)*10))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is much drier than the original example, and is more code for the reader to read and understand, but that code and the knowledge gained from understanding it is a distraction from the point of the example rather than an enhancement. So again I prefer the original here.

It also seems to me, looking at this original example, that it is already demonstrating the multi-line behavior that you are saying isn't covered. x is set in one test, and then interrogated in two subsequent tests. Your revision does make that point more strongly, however,

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to a better example.
I was trying to demonstrate both a compound statement and a multi-line output string, as well as being clear about setting state in one statement and testing it in the next. What I came up with is pretty contrived.
Maybe have the test be a while loop?
Frankly, I didn't find the if… print("yes")… print("NO!!!") example all that impressive. It seemed even more contrived, and silly rather than funny. Maybe others appreciate it more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multi-line examples having a setup phrase is already covered in the original example as @bitdancer said, in:

>>> x = 12
>>> x
12

In your example, the line: >>> x = factorial(10); math.ceil(math.log10(x)) is not clear, I do not expect one to know the REPL is printing the last statement of a line containing multiple ones separated by semi columns, it just make the example hard to read, if not surprising. Yes I even had to test it in my REPL to convince me it actually works.

Please do not change the example, it was simple and covered the point you're trying to fix.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JulienPalard , thank you for your review.
The core of this PR is that I wrote an "example" which required an import statement to run, and this documentation didn't explain clearly to me how I could do this right. So, for me as a reader, the existing text was not sufficient:

>>> x = 12
>>> x
12

And the core of what I am proposing is to use a different demonstration of multiple "examples" which build up state and then use it, as in:

>>> import math
>>> x = factorial(10)
>>> math.ceil(math.log10(x))
7

(I believe there is a benefit of using a statement group with ; instead of simple statements, because it obliquely explains one more aspect of how the doctest feature works, but that is a side issue.)

You said, "Please do not change the example, it was simple and covered the point you're trying to fix." My feedback to you is that, as a developer and a reader of this documentation, it did not cover the point. It did not tell me how to use use an import statement in my "example". The core of this PR is to make documentation do a better job of covering this point.

...
no
NO
NO!!!
same:
3628800
>>>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By deleting this intro, you've lost the context in which the subsequent detailed explanation is given. However, this existing intro doesn't give quite enough information either. I suggest:

A doctest example is composed of one or more tests. An individual test starts with a line that starts with '>>>', has zero or more code continuation lines that start with '...', and ends with zero or more expected output lines. The expected output ends at the first line that starts with '>>>' or is blank. All lines in an example block must have the same indentation level.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Funny, you describe the deleted paragraph as an "intro". I looked at it as the lagging statement of "this is what doctest does", which should have been before the example.
Your suggested paragraph is quite good. how about moving that to the above the example, replacing my intro paragraph?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, the latest version does take @bitdancer's intro and moves it to above the example, replacing my original intro paragraph. The Github discussion doesn't make that clear.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As in a previous review I asked you to remove the new intro paragraph with wrong statement in it, please keep this one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, @JulienPalard , I'm having a hard time finding the review where you ask me to remove the "new intro paragraph with wrong statement in it". I don't know how to respond to this feedback.


Any expected output must immediately follow the final ``'>>> '`` or ``'... '``
line containing the code, and the expected output (if any) extends to the next
``'>>> '`` or all-whitespace line.

The fine print:

* The >>> marks the start of an interactive statement: that is, a
statement list ending with a newline, or a :ref:`compound statement <compound>`.
The ... string indicates that the line continues a compound statement.

* If the expected output is empty, it indicates that the test generates
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation use the word example, not test, please use it here too:

If the expected output is empty, it indicates that the example generates no output when run.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that Doc/library/doctest.rst is inconsistent in how it uses the words "example", "test", and "docstring". I was trying to make the wording consistent. But this isn't the heart of what I want to do with this P.R. Will you accept it if I make each paragraph keep using the same word "test" or "example" that it currently uses?

no output when run. If the test does generate output, the module reports
it as a failure.

* The expected output can contain multiple lines. These lines become a
string containing newlines. The leading indentation of the example
block is stripped when building the string. The resulting string is
compared to the string of actual output from running the test.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*example


* The last code continuation line of an example copied from the
interactive shell (the line starting with "..." that is otherwise
blank in the example above) may be omitted without changing the
meaning of the test.

* Expected output cannot contain an all-whitespace line, since such a line is
taken to signal the end of expected output. If expected output does contain a
blank line, put ``<BLANKLINE>`` in your doctest example each place a blank line
blank line, put ``<BLANKLINE>`` in your doctest test each place a blank line
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree, the doctest module is not aimed to write tests in docstrings, but to test examples found in docstrings. So here example was the right word.

is expected.

* All hard tab characters are expanded to spaces, using 8-column tab stops.
Expand Down Expand Up @@ -373,21 +392,28 @@ The fine print:
1

and as many leading whitespace characters are stripped from the expected output
as appeared in the initial ``'>>> '`` line that started the example.
as appeared in the initial ``'>>> '`` line that started the test.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same.



.. _doctest-execution-context:

What's the Execution Context?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

By default, each time :mod:`doctest` finds a docstring to test, it uses a
*shallow copy* of :mod:`M`'s globals, so that running tests doesn't change the
module's real globals, and so that one test in :mod:`M` can't leave behind
crumbs that accidentally allow another test to work. This means examples can
freely use any names defined at top-level in :mod:`M`, and names defined earlier
in the docstring being run. Examples cannot see names defined in other
docstrings.
Within a docstring, later tests can use names defined by earlier
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not say so, exemples should be as autonomous as possible to make sense, one should not write an example containing only setup statements, then other examples using them, it's possible, but it's not a feature, just a bad practice, let's not encourage it. Still one can put setup statement in its example so the example is reproductible by itself (autonomous). Please remove this paragraph.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JulienPalard, the core of this PR is that I wanted to test some code which required an import of another module to run. Is this "bad practice"? If I think I need to import a module to make an "example" run, should I stop using doctests and put that code in a unittest fixture instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a bad practice to use import in a doctest, I'm saying you don't need two separate doctests in the same docstring to do so, a single doctest is enough, like:

"""
An example:
>>> import math
>>> math.sqrt(4)
2.0
"""

I do not even consider bad practice separating imports and examples, like:

"""
An example:
First do the import:
>>> import math

Then use sqrt:
>>> math.sqrt(4)
2.0
"""

Yet it's harder to read, it's visually two distincts sessions, with a lot of context switch (text, repl text, repl). It would make more sense when demoing 4 different features, avoiding to import 4 times.

The point of my comment: you're stating "Within a docstring, later tests can use names defined by earlier" which is generalizing to variables, not only imports, and I see very few examples where it can be readable with shared variables. After reading this line, I image one writing:

def foo():
    """
    First example with division:
    >>> a = 10 / 3
    >>> print(a)
    3.3333333333333335

    Second example, with floor division:
    >>> a // 2
    1.0
    """

Which is just bad (this is what I consider "bad practice"): the second example will work under doctest, but one every other user will probably just be interested by the second example, which will fail if they try it alone. This is why I'm speaking of "autonomous examples".

examples. It's fine for a test to set up state, and
have no output.

For each docstring, :mod:`doctest` makes (by default) a
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're reformulating this paragraph but it is not linked to the original issue you're trying to fix.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JulienPalard , thank you for your review.

Maybe this is a philosophical issue about documentation. I believe documentation works as a related structure: a particular idea may be introduced early in the document, illustrated with examples in the middle of the document, and then defined with detailed text later in the document. So, improving how a document describes one concept may touch multiple places in a file. I worry that all the insistence on limiting changes just a few adjacent lines makes it hard to refactor the concepts in a document. It also makes it harder to review how a set of changes to the same document will affect the coherence of that document.

Reformulating this paragraph is related to the original issue I'm trying to fix. The original issue is how an import statement is handled in doctests. This is related to concepts of how the state accumulates across examples within a docstring, and how state is reset between docstrings. This paragraph does a poor job of describing how state is reset between docstrings. That is why I try to improve it.

But if it's not possible to refactor documents, only to propose changes to one paragraph or another, then I can break this P.R. up into multiple, each trying to fix one weakness. I think that will be harder to review, and more likely to result in partial changes that are incoherent. But I will try to improve what I can improve.

*shallow copy* of :mod:`M`'s globals. This means tests can freely
use any names defined at the top level of :mod:`M`.
When doctest performs tests, it doesn't change the module's real globals.

This shallow copy of globals is discarded after the docstring has been
processed, and copied afresh for the next docstring. Thus, tests in one
docstring in :mod:`M` can't leave behind crumbs that accidentally allow an
test in another docstring to work. Tests cannot see names defined in
other docstrings.

You can force use of your own dict as the execution context by passing
``globs=your_dict`` to :func:`testmod` or :func:`testfile` instead.
Expand Down