Multi-part examples in rustdoc

[mightyiam]

@arifd

Co-authored-by: Arif Driessen [email protected]

[GuillaumeGomez]

I personally don't think this is a needed addition: it seems to be a bit too much specific to be really useful (or used).

I'm interested to see what the others in the @rust-lang/rustdoc team think about it though.

[camelid]

How would you indicate that? split-start,rust or rust,split-start or something else?

[jyn514]

Code blocks are rust by default.

[camelid]

I mean rust as a generic language tag — I could have been clearer by saying split-start,javascript. I think I feel a bit confused about why it says "Not only rust examples may be multi-part."

[mightyiam]

Does 592ede1 resolve your concerns?

[camelid]

What does this mean?

[mightyiam]

In 7e6c746 we tried to clarify.
In 524dc64 we removed a paragraph in which we meant to express our prediction that this feature will be used and even used if it is behind a feature flag.

[jyn514]

Doctests are pretty common.

• Python
• Ruby
• Elixir
• D
• Racket has runtime contracts that are part of the documentation: https://blog.racket-lang.org/2012/06/submodules.html#in-source-documentation.

It would be good to know if any of them have interleaved tests like this.

[mightyiam]

Thank you for the links to doc-tests in other languages. Actually, our paragraph wasn't clear.
We know that doc-tests is a common feature.
What we were trying to say is that we're not aware of are features in other languages that are equivalent to the one we are suggesting in this RFC.
Corrected with 9992d9f.

We hope that the community could refer us to any of those.
Note that for an equivalent feature to be really equivalent, it needs to:

1. Generate readable documentation, such as HTML from source code documentation
2. Extract tests from examples in that source code documentation
3. Support some rich text format interleaved with code lines of those examples

[mgeisler]

It would be good to know if any of them have interleaved tests like this.

This is supported in Python where you can write a docstring like this:

"""
First code block:

>>> x = 10

Second code block:

>>> x
10
"""

[jyn514]

I personally don't think this is a needed addition: it seems to be a bit too much specific to be really useful (or used).

I think this is useful, although it does stretch the scope of rustdoc a bit. I'd like to hear from library authors, since they'll be the ones using it - @Darksonn @BurntSushi @dtolnay would you be interested in this?

[Darksonn]

I do sometimes write things in this style, but I have not found the need to do so in our API reference. The examples there are not really complicated enough to warrant it in my experience.

Maybe it would be more useful in mdBook?

[alice-i-cecile]

I would love to have this for Bevy! Actually demonstrating functionality can involve quite lengthy, multipart code, and we rely heavily on our stand-alone and docs-integrated examples as integration tests.

[mightyiam]

@jyn514 continuing from #3081 (comment) we were on the fence about this. I, personally, don't like the feature of rust being default, for the same compatibility concern. I guess that's where I am coming from in this.

We wouldn't mind if it is deemed more desirable that subsequent code blocks inherit the language of the first one.
If you ask me, I feel it is consistent with a design feature I don't agree with, making things only slightly worse.

Another point is that this inheritance would take a little bit more implementation work, wouldn't it? Not sure it's a relevant point or not. If there is difference in implementation, perhaps we could first error and then re-visit this inheritance later, if anyone really cares.

[mightyiam]

Also continuing #3081 (comment).

[jyn514]

This has two different 'Run' buttons, one on each code block. Would that run the full example in both cases? If not, does it make sense to only add it to the second block?

[mightyiam]

Now that I have looked into this RFC more, I am not certain about a very relevant point: those run buttons — where do I know them from? Where are they currently implemented? Is it in rustdoc or in mdBook? The only place I have certainly seen them is the standard library documentation.

This point is relevant because if they are not a rustdoc feature, then it would seem of no concern to this RFC and the mockup should not include such buttons.

[bjorn3]

It is part of rustdoc. I think you need the #![doc(html_playground_url)] attribute for them to show up: https://github.com/rust-lang/rust/blob/4c55c8362de32be1530b2441c3e3a51e73edeb21/library/core/src/lib.rs#L58

[camelid]

It's also implemented in mdBook, but as a button that looks (roughly) like this: ⏯️

[steveklabnik]

I wonder what @carols10cents thinks of this RFC. I have not read it yet, but this feature in general feels like something that we may want to take advantage of in the book. I know at various times I've wanted something like this in long-form writing about Rust.

[lebensterben]

This is a really nice-to-have feature.
I especially like the idea of adding arguments to ``` for special behaviors, which resembles RMarkdown.

[camelid]

I have not read it yet, but this feature in general feels like something that we may want to take advantage of in the book. I know at various times I've wanted something like this in long-form writing about Rust.

Doesn't the Book use mdBook though?

[steveklabnik]

@camelid unless something has changed (or i am 1000% misremembering), mdbook calls out to rustdoc.

[camelid]

@camelid unless something has changed (or i am 1000% misremembering), mdbook calls out to rustdoc.

Ah, I forgot that mdBook uses rustdoc for tests 👍

[carols10cents]

So yes, the book does use mdBook, and mdBook takes Rust code blocks from the markdown files and then runs rustdoc test on them. So it's kind of backwards from library documentation.

When we have long examples in the book that we explain in pieces, I'm using the rustdoc_include feature with anchors. The code actually lives in separate files so that we can provide complete, standalone Cargo projects for each code listing (and can run rustfmt on them, save the output of running them to include in the text as well, and do other stuff that I doubt is relevant to library documentation).

I believe there are a few cases where one code file has multiple anchors in it, so that we are accomplishing the same goal that is here of having text interleaved with multiple parts of one long code example. But most of the time, in the book, we're helping the reader to build up an example from nothing, and it's not often straightforward like "add lines 3 and 4, then add lines 5, 6, and 7". Usually it's "we're going to write a function that returns a static value, ok now we're going to take an argument, ok now we're going to change the return type, and now we have to change the call sites too". So it's more applying changes to one example to build it up, rather than explaining it line-by-line.

So I don't think this feature as written would be relevant to the book, is what I'm trying to get at.

Now, for library documentation, I personally find it very difficult to write long doc comments anyway, to get them to show up where I think people would look for them in the moment they need them. And writing markdown inside /// lines isn't a great experience, writing Rust code within Markdown within doc comments even more so. If I provide an example in a doc comment, it's going to be very short and something to just give the general idea of how you'd call something as reference, not a tutorial with lots of text around it. I think mdBook or other tools are better for the tutorial kind of documentation, at this point (I'm not sure what the best tool for writing tutorials is, but I'm familiar with mdBook and it integrates with Rust code well enough, so it's what I'd currently use in this case).

So overall, I'm not sure that adding this feature to rustdoc would be useful. I do, however, appreciate the thought that has gone into this RFC and appreciate the desire to make docs better!

[BurntSushi]

As a library author that writes lots of examples, I'm not sure I would have much use for something like this personally. While I'm not quite as pessimistic about writing code inside of /// comments as @carols10cents (perhaps because I've done it so much that my brain has adapted to it haha), I do not often find myself in situations where I have drawn out examples that would require something like this. In cases where I do, I often find it better to find a way to simplify the example somehow. When that's not possible, putting comments inside the code block is usually good enough for me. It's likely I would continue doing that in most cases even if this feature were present, since I think splitting up examples into multiple chunks can be a bit disorienting unless the situation really calls for it. But maybe this is a feature where if it were present, I'd find more use cases for it.

With respect to the UI mockup, I have two comments/questions:

1. What does the "run" button do on each of the snippets? I would guess that the "run" button does the same thing for each snippet in an entire example: it runs the whole thing. Where does the output get displayed?
2. Is there a way to copy the entire code block stitched together? Otherwise, users will have to copy each individual block if they want to copy an example into their project.

[shepmaster]

Prompted by @carols10cents' comment, I realized that I would be much more inclined to write longer form documentation¹ if the effort involved with publishing a book was reduced. To that end, I'm suggesting that docs.rs build and host mdBook content. That may be a complete alternative to this RFC.

---

1 — And I already try to write longer form documentation inside of my more user-facing crates. For example, SNAFU has an entire guide module that is only documentation.

[mgeisler]

Python has a long history of using doctests: comments with embedded Python interpreter sessions. They look like this:

This is an example text file in reStructuredText format.  First import
``factorial`` from the ``example`` module:

    >>> from example import factorial

Now use it:

    >>> factorial(6)
    120

Notice how the two >>> lines share state: they're part of one interpreter session. The example is from the doctest documentation. This style of comment was supported at least 20 years ago as seen in PEP-287.

The Python doctest library has been ported to other languages, such as Haskell. The documentation for that library explains that it clears the state between each code block by default (but there is a --fast flag which disables this since there is a performance cost to clearing the state).

[beeryt]

Why don't we just concatenate code blocks within a single doc comment?
It seems like most code blocks within a doc comment are in the context of the item being documented.

/// We can bind a value to a variable
/// ```
/// let x = 0;
/// ```
/// We can use the bound variable as a value in another binding
/// ```
/// let y = x;
/// ```
/// We can use `shadowing` to bind a new variable with the name of an existing variable
/// ```
/// let x = 5;
/// ```
/// Note that the original variable `x` is still in scope, but can no longer be directly accessed

This would compile to:

let x = 0;
let y = x;
let x = 5;

For the most part, it seems variable shadowing can easily be used to soft-reset variable names without resetting the entire scope. And if users are really particular about their scope they can be explicit with braces (with or without #).

/// ```
/// # {
/// let x = 0;
/// ```
/// ```
/// let y = x;
/// # }
/// ```
/// ```
/// let x = 5;
/// ```

Are there prior discussions with reasoning for avoiding this?

[mightyiam]

Thank you for the suggestion, @beeryt. Do you feel that implicit concatenation of code blocks fits in with other design decisions in Rust?

For one, that would be a breaking change.

[beeryt]

I'm not familiar with prior design decisions, but here is an addendum to my suggestion to possibly avoid a breaking change:

Add a #[doc] attribute like test(combine_code_blocks).

#[doc(test(combine_code_blocks))]
/// ```
/// let x = 0;
/// ```
/// ```
/// let y = x;
/// ```

I'm not sure whether this would be better as an item-level (#[doc...]) or crate-level (#![doc...]).

[GuillaumeGomez]

Combining by default would break std and quite a huge amount of crates. I don't think an attribute is the right way to do it either. Using codeblock labels seems like a better idea:

/// ```combine
/// let x = 0;
/// ```
///
/// blabla
///
/// ```
/// let x = 1;
/// ```
///
/// ```combine
/// assert!(x == 0);
/// ```

[GuillaumeGomez]

You mentioned yourself that an attribute would break std docs. So from that, I don't think this approach is worth it.

[beeryt]

It would only break std if the attribute was added to std, right? I was thinking this was an approach which would avoid breaking existing docs.

[GuillaumeGomez]

True. Sorry. I badly exposed what I had in mind. What I meant is that if you opt in this attribute, either on an item or not, then you remove the possibility to have different examples and combined examples in the same docs.

[beeryt]

Maybe a combination of my attribute and codeblock labels could work. I'm just brainstorming ideas that are ideally less verbose in both scenarios. I imagine it is less common for someone to explicitly want separate scopes for every code block while also wanting some separation.

#[doc(test(combine_code_blocks))]
/// ```
/// let x = 0;
/// ```
/// ```
/// let y = x;
/// ```
/// ```new_scope
/// let x = 5;
/// ```
/// Below will `panic!`
/// ```
/// assert_eq!(x, 0);
/// ```

Maybe (and I don't know if the label syntax would support this) allowing us to set named scopes could work:

#[doc(test(combine_code_blocks))]
/// ```
/// let x = 0;
/// let y = x;
/// ```
/// ```set_scope(name_or_index)
/// let x = 5;
/// ```
/// Below will pass because it is part of the *default* scope
/// ```
///  assert_eq!(x, 0);
/// ```

This example would create two scopes for execution:

// *default* scope
{
  let x = 0;
  let y = x;
  assert_eq!(x, 0);
}

// scope(name_or_index)
{
  let x = 5;
}

[mgeisler]

Why don't we just concatenate code blocks within a single doc comment?
It seems like most code blocks within a doc comment are in the context of the item being documented.

Yes, I had the same idea — and I think it would work very well. The stdlib examples show that it sometimes won't work, but from my experience, it's very rare to include full main functions in doctests, especially since the point of a doctest is to be lightweight.

My proposal would be to implement it conservatively by concatenating the code blocks per item. If this doesn't compile, then try again without concatenating.

That would allow all code to keep working.

We could perhaps remove the fallback in a new edition? I know the standard library has to be compatible across editions, but we could just rewrite the doctests to avoid the breakage.

As alternative, we could have people mark code blocks now if they want them to be concatenated and then make this a default later.

[mgeisler]

think this is useful, although it does stretch the scope of rustdoc a bit. I'd like to hear from library authors, since they'll be the ones using it - @Darksonn @BurntSushi @dtolnay would you be interested in this?

I've been missing this a lot: I come from Python where doctests are widely supported. Doctests in Python allow you to save state between code blocks and this is very useful to be able to show a continued worked examples. Here is a random example from Scipy where they first define a matrix and then verify some property of it in the next code block:

Note how the second code block uses a defined in the first. I hope to write similar documentation in Rust where I only need to import modules once per docstring.