Improve Notebook Output Rendering #243

chrisjsewell · 2020-08-23T07:54:10Z

No description provided.

- Ensure source (path, lineno) are correctly propagated to `CellOutputBundleNode` - Capture cell level metadata in `CellOutputBundleNode` - New `CellOutputRenderer` class to contain render methods - Simplify test code, using sphinx `get_doctree` and `get_and_resolve_doctree` methods

This allows ofr other post-transforms, like `ReferencesResolver` to act on the output nodes.

codecov · 2020-08-23T07:56:39Z

Codecov Report

Merging #243 into master will increase coverage by 0.64%.
The diff coverage is 88.01%.

@@            Coverage Diff             @@
##           master     #243      +/-   ##
==========================================
+ Coverage   86.52%   87.16%   +0.64%     
==========================================
  Files          10       12       +2     
  Lines         987     1239     +252     
==========================================
+ Hits          854     1080     +226     
- Misses        133      159      +26

Flag	Coverage Δ
#pytests	`87.16% <88.01%> (+0.64%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
myst_nb/__init__.py	`88.27% <82.60%> (-1.33%)`	⬇️
myst_nb/ansi_lexer.py	`84.70% <84.70%> (ø)`
myst_nb/render_outputs.py	`86.89% <86.89%> (ø)`
myst_nb/nb_glue/domain.py	`90.45% <100.00%> (-0.19%)`	⬇️
myst_nb/nb_glue/transform.py	`82.35% <100.00%> (ø)`
myst_nb/nodes.py	`100.00% <100.00%> (ø)`
myst_nb/parser.py	`96.40% <100.00%> (-0.31%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b09901...3368a64. Read the comment docs.

The notebook renderer class is now loaded from an entrypoint, with a configurable name. Also moved node classes to a separate module.

ANSI lexer is applied to stdout/stderr and text/plain by default

mgeier · 2020-08-23T16:50:12Z

If you need a few test cases for ANSI coloring, feel free to have a look at https://nbsphinx.readthedocs.io/code-cells.html#ANSI-Colors

I've implemented the ANSI support in nbconvert, notebook and jupyterlab, so I know a few of the pitfalls.
If you need pointers, please let me know.

chrisjsewell · 2020-08-23T18:42:44Z

Thanks @mgeier!
I added your 8-bit ANSI into the docs, and it looks to be working well 😄 https://myst-nb--243.org.readthedocs.build/en/243/use/formatting_outputs.html#ansi-outputs
I'll leave 256-but ANSI for another time/PR though (or feel free to give it a go 😬)

chrisjsewell · 2020-08-23T21:48:44Z

Hey @mmcky and @AakashGfude (and I see you lurking @choldgraf 👀 😆) I have maybe a few more bits to add on this, but the main restructuring and features are there if you want to have a look and give any thoughts.
The commit messages and this page should explain the main changes/additions: https://myst-nb--243.org.readthedocs.build/en/243/use/formatting_outputs.html

chrisjsewell · 2020-08-23T22:03:40Z

Oh and also @akhmerov; a) because I'm sure it'll be of interest, but also b) because you can now close jupyter/jupyter-sphinx#130 however you see fit, since I don't even use cell_output_to_nodes any more lol

akhmerov · 2020-08-23T22:19:02Z

@chrisjsewell thanks for the heads up, this is indeed very informative. At a glance render_outputs.py seems like it could as well be in jupyter_sphinx. Did you consider that option?

chrisjsewell · 2020-08-23T22:33:32Z

Well we can have a look and think about that at a later date, but for now (a) its too much complication, in terms of cross-repo testing, documentation, reviewing, etc, but also (b) it has dependencies on myst-parser, myst-nb configuration values, and relies on the use of cell metadata, which jupyter-sphinx's directives might have difficulty handling.

So I'm open to the possibility, but I'm afraid I haven't really got time to figure out how 😬

akhmerov · 2020-08-23T22:39:30Z

Sounds reasonable. As a quick note: factoring out myst-specifc parts would amount to making an entry point for the myst-markdown renderer.

Passing render state via cell tags seems alright.

chrisjsewell · 2020-08-23T22:45:37Z

Passing render state via cell tags seems alright.

Well, its going to be more than just tags...

akhmerov · 2020-08-23T22:47:57Z

Really? That seemed to be the only context where metadata occurs in render_outputs.py.

chrisjsewell · 2020-08-23T22:58:31Z

Really? That seemed to be the only context where metadata occurs in render_outputs.py.

Thats now, in a few hours, maybe not lol

chrisjsewell · 2020-08-24T03:00:51Z

@akhmerov now this is what I'm talking about 😄 https://myst-nb--243.org.readthedocs.build/en/243/use/formatting_outputs.html#images

mmcky · 2020-08-24T03:23:15Z

docs/use/formatting_outputs.md

+We can also set a caption (which is rendered as [CommonMark](https://commonmark.org/)) and name, by which to reference the figure:
+
+````md
+```{code-cell} ipython3


@chrisjsewell what happens if the code in this code-block doesn't produce an image? Do we throw an error if the returned mime type of the output block isn't compatible with the image directive?

Then the render_image method is never called, and the metadata just isn't used,
i.e. the metadata is accessed (and validated) "lazily" when it is required

If it is invalid, for example I give a bad width here, then it produces a warning, and since this a text-based notebook, that warning will point to the correct line of the document (the first one of the code cell)

```{code-cell} ipython3 --- myst: image: width: abc --- from IPython.display import Image Image("images/fun-fish.png") ```

/Users/chrisjsewell/Documents/GitHub/MyST-NB-actual/docs/use/formatting_outputs.md:110: WARNING: output render: Invalid image attribute: (key: 'width'; value: abc) not a positive measure of one of the following units: "em" "ex" "px" "in" "cm" "mm" "pt" "pc" "%"

and also, because the image is no longer created:

/Users/chrisjsewell/Documents/GitHub/MyST-NB-actual/docs/use/formatting_outputs.md:124: WARNING: 'myst' reference target not found: fun-fish

I see -- thanks @chrisjsewell

mmcky

thanks @chrisjsewell. It is neat to be able to apply styles and formatting on images that are generated by code and the ansi support is timely on the LaTeX side.

For ansi support -- I came across the ansifilter project this morning that may be useful in converting to various formats. However I don't think it is directly importable to python that I can tell.

For LaTeX output -- from what I can tell reading the code you parse ansi codes and return a new token object for inclusion into the AST so we just need to support those new node types when writing the tex (or skip those elements for stripping output of ansi codes)?

mmcky · 2020-08-24T03:39:24Z

docs/use/formatting_outputs.md

+      "KL\x1b[49mMN\x1b[39mOP\x1b[22mQR\x1b[24mST\x1b[27mUV")
+```
+
+This uses the built-in {py:class}`~myst_nb.ansi_lexer.AnsiColorLexer` [pygments lexer](https://pygments.org/).


@chrisjsewell is there a reason we don't just import this from pygments.

Because its not part of pygments, there is no built-in ANSI lexer unfortunately (from what I could see), this is a bespoke one

It appears to work well for the 8-bit ansi though, perhaps some finessing of the CSS.
Then, as it says in the docs, we can get round to the adding 256-bit support as and when necessary

Oh OK -- sorry had misread header comment on the file. Makes sense.

Is this something we should contribute upstream to pygments at some point?

Yeh quite possibly; just wanted to get everything working first, and then we can look to tidy up

chrisjsewell · 2020-08-24T03:41:11Z

that may be useful in converting to various formats

Well, fingers crossed, that should already happen, because we are using pygments to lex it to tokens, then sphinx chooses the appropriate formatter to convert it: https://pygments.org/docs/formatters/

chrisjsewell · 2020-08-24T04:50:31Z

@mmcky has given i the thumbs up for "working" (or at least not breaking) LaTeX, so I'm going to merge this monstrosity!

mmcky · 2020-08-24T04:52:32Z

docs/use/formatting_outputs.md

+You can change the lexer used in the `conf.py`, for example to turn off lexing:
+
+```python
+nb_render_text_lexer = "none"


@chrisjsewell one last question. Does this option in conf.py just control ansi parsing of text?

yep, it controls what pygments lexer is applied to plain text outputs (including stdout/stderr)

ok - so it does more than just ansi. Was just thinking it might be better named nb_render_text_ansi but it is more generally a control over the lexer used which contains more than just ansi parsing.

mgeier · 2020-08-24T07:22:01Z

I added your 8-bit ANSI into the docs, and it looks to be working well

It doesn't seem to support "underline" and "invert", and there don't seem to be any "intense" colors when switching to "bold".

I'll leave 256-but ANSI for another time/PR though (or feel free to give it a go)

I've already implemented this in three different places (in two languages), and I don't currently have the energy to do it yet again.

Do you even have to use Pygments? Why not just use nbconvert's implementation?

For reference, the nbconvert implementation is there: https://github.com/jupyter/nbconvert/blob/master/nbconvert/filters/ansi.py

chrisjsewell added 4 commits August 23, 2020 07:29

✨ NEW: Allow configuration of render priority

beaa7a5

✨ NEW: Add remove-stderr and remove-stdout metadata tags

d188465

👌 IMPROVE: Increase priority of Nb output render

5522770

This allows ofr other post-transforms, like `ReferencesResolver` to act on the output nodes.

This was linked to issues Aug 23, 2020

Bug: Can't render the tex output of julia documents #183

Closed

Figures from notebook outputs with link to img file #240

Closed

"filter" unwanted execution outputs (e.g. stdout/stderr) #228

Closed

♻️ REFACTOR: make Nb output renderer pluggable

0646b11

The notebook renderer class is now loaded from an entrypoint, with a configurable name. Also moved node classes to a separate module.

chrisjsewell mentioned this pull request Aug 23, 2020

[ENH] Added Builder and execute options for page and added execution timeout key jupyter-book/jupyter-book#744

Merged

✨ NEW: Nb outputs ANSI lexer

0e05a19

ANSI lexer is applied to stdout/stderr and text/plain by default

✨ NEW: render text/markdown MIME type

a0af450

chrisjsewell mentioned this pull request Aug 23, 2020

render toc paths as posix strings for windows jupyter-book/jupyter-book#723

Closed

chrisjsewell added 2 commits August 23, 2020 20:55

📚 DOCS: render priority & stdout/stderr removal

42fad19

📚 DOCS: API and custom render classes

c4f3b40

chrisjsewell force-pushed the improve/cell-metadata branch from 73098ec to c4f3b40 Compare August 23, 2020 21:44

This was referenced Aug 24, 2020

✨ NEW: Custom notebook formats #242

Merged

Support for ANSI color sequences in Jupyter cell output jupyter-book/jupyter-book#762

Closed

✨ NEW: Add code output image formatting

06f8ae5

chrisjsewell marked this pull request as ready for review August 24, 2020 03:01

chrisjsewell mentioned this pull request Aug 24, 2020

ENH: Enable auto-configuration to build all content pages as individual PDF files jupyter-book/jupyter-book#687

Closed

mmcky reviewed Aug 24, 2020

View reviewed changes

👌 IMPROVE: minor formatting and test additions

3368a64

mmcky reviewed Aug 24, 2020

View reviewed changes

mmcky mentioned this pull request Aug 24, 2020

[ENH] PDF theme jupyter-book/jupyter-book#586

Closed

chrisjsewell merged commit 04f3bbb into master Aug 24, 2020

chrisjsewell deleted the improve/cell-metadata branch August 24, 2020 04:57

chrisjsewell mentioned this pull request Aug 24, 2020

Code Cell Output Improvements (labels, captions, ...) #72

Closed

chrisjsewell mentioned this pull request Aug 24, 2020

Improve ANSI lexing #247

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve Notebook Output Rendering #243

Improve Notebook Output Rendering #243

chrisjsewell commented Aug 23, 2020

codecov bot commented Aug 23, 2020 •

edited

Loading

mgeier commented Aug 23, 2020

chrisjsewell commented Aug 23, 2020

chrisjsewell commented Aug 23, 2020 •

edited

Loading

chrisjsewell commented Aug 23, 2020 •

edited

Loading

akhmerov commented Aug 23, 2020

chrisjsewell commented Aug 23, 2020 •

edited

Loading

akhmerov commented Aug 23, 2020 •

edited

Loading

chrisjsewell commented Aug 23, 2020

akhmerov commented Aug 23, 2020 •

edited by chrisjsewell

Loading

chrisjsewell commented Aug 23, 2020

chrisjsewell commented Aug 24, 2020

mmcky Aug 24, 2020

chrisjsewell Aug 24, 2020

chrisjsewell Aug 24, 2020

mmcky Aug 24, 2020

mmcky left a comment •

edited

Loading

mmcky Aug 24, 2020 •

edited by chrisjsewell

Loading

chrisjsewell Aug 24, 2020 •

edited

Loading

chrisjsewell Aug 24, 2020

mmcky Aug 24, 2020 •

edited

Loading

mmcky Aug 24, 2020

chrisjsewell Aug 24, 2020 •

edited

Loading

chrisjsewell commented Aug 24, 2020

chrisjsewell commented Aug 24, 2020

mmcky Aug 24, 2020 •

edited

Loading

chrisjsewell Aug 24, 2020

mmcky Aug 24, 2020

mgeier commented Aug 24, 2020

Improve Notebook Output Rendering #243

Improve Notebook Output Rendering #243

Conversation

chrisjsewell commented Aug 23, 2020

codecov bot commented Aug 23, 2020 • edited Loading

Codecov Report

mgeier commented Aug 23, 2020

chrisjsewell commented Aug 23, 2020

chrisjsewell commented Aug 23, 2020 • edited Loading

chrisjsewell commented Aug 23, 2020 • edited Loading

akhmerov commented Aug 23, 2020

chrisjsewell commented Aug 23, 2020 • edited Loading

akhmerov commented Aug 23, 2020 • edited Loading

chrisjsewell commented Aug 23, 2020

akhmerov commented Aug 23, 2020 • edited by chrisjsewell Loading

chrisjsewell commented Aug 23, 2020

chrisjsewell commented Aug 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmcky left a comment • edited Loading

Choose a reason for hiding this comment

mmcky Aug 24, 2020 • edited by chrisjsewell Loading

Choose a reason for hiding this comment

chrisjsewell Aug 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmcky Aug 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisjsewell Aug 24, 2020 • edited Loading

Choose a reason for hiding this comment

chrisjsewell commented Aug 24, 2020

chrisjsewell commented Aug 24, 2020

mmcky Aug 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgeier commented Aug 24, 2020

codecov bot commented Aug 23, 2020 •

edited

Loading

chrisjsewell commented Aug 23, 2020 •

edited

Loading

chrisjsewell commented Aug 23, 2020 •

edited

Loading

chrisjsewell commented Aug 23, 2020 •

edited

Loading

akhmerov commented Aug 23, 2020 •

edited

Loading

akhmerov commented Aug 23, 2020 •

edited by chrisjsewell

Loading

mmcky left a comment •

edited

Loading

mmcky Aug 24, 2020 •

edited by chrisjsewell

Loading

chrisjsewell Aug 24, 2020 •

edited

Loading

mmcky Aug 24, 2020 •

edited

Loading

chrisjsewell Aug 24, 2020 •

edited

Loading

mmcky Aug 24, 2020 •

edited

Loading