Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add notes on error-tolerant execution to CLI docs #972

Merged
merged 3 commits into from
Jul 4, 2022
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 31 additions & 12 deletions docs/using-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,6 @@ jupytext --set-formats ipynb,py notebook.ipynb # Turn notebook.ipynb into a pai
jupytext --sync notebook.ipynb # Update whichever of notebook.ipynb/notebook.py is outdated
```

For convenience, when creating a notebook from text you can execute it:
```bash
jupytext --set-kernel - notebook.md # create a YAML header with kernel metadata matching the current python executable
jupytext --set-formats md:myst notebook.md # create a YAML header with an explicit jupytext format
jupytext --to notebook --execute notebook.md # convert notebook.md to an .ipynb file and run it
```

If you wanted to convert a collection of Markdown files to paired notebooks, and execute them in the current Python environment, you could run:
```bash
jupytext --set-formats ipynb,md --execute *.md
```

You may also find useful to `--pipe` the text representation of a notebook into tools like `black`:
```bash
jupytext --sync --pipe black notebook.ipynb # read most recent version of notebook, reformat with black, save
Expand All @@ -58,6 +46,37 @@ Note also that on Windows you need to use double quotes instead of single quotes

Execute `jupytext --help` to access the full documentation.

### Execute notebook cells

For convenience, when creating a notebook from text you can execute it:

```bash
jupytext --set-kernel - notebook.md # create a YAML header with kernel metadata matching the current python executable
jupytext --set-formats md:myst notebook.md # create a YAML header with an explicit jupytext format
jupytext --to notebook --execute notebook.md # convert notebook.md to an .ipynb file and run it
```

If you wanted to convert a collection of Markdown files to paired notebooks, and execute them in the current Python environment, you could run:
```bash
jupytext --set-formats ipynb,md --execute *.md
```

Please note: if any notebook cell errors, execution will terminate and `jupytext` will not save the notebook. This can cause headaches as the details of any error would be encoded in the notebook that isn't saved. But there's a way: `jupyter nbconvert` has a mode which will still save a notebook if a cell errors, producing something akin to what would happen if you ran all cells manually in Jupyter's notebook UI.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that having the notebook fail is an unpleasant experience, yet failing on error is also nbconvert's standard, so I'd rather introduce the next paragraph as something like advanced notebook execution. What do you think?


```bash
# First, convert script (py/sh/R/jl etc) -> notebook. May need additional args to define input format etc as above.
jupytext --to ipynb script.sh
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this but I am afraid most users convert .py scripts to notebooks... maybe the first example can take a .py script, and then recall that .sh scripts can also be converted to Bash notebooks (do you execute them with bash_kernel by the way?)

Copy link
Contributor Author

@kdm9 kdm9 Jul 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll s/.sh/.py then :)

Also, yes, with bash_kernel (though it seems the latest release & HEAD are broken, see takluyver/bash_kernel#120)

# Then, execute notebook in place and allowing cells to produce errors
jupyter nbconvert --to ipynb --inplace --execute --allow-errors script.ipynb
# One can also combine these to a single command using jupytext --pipe
jupytext --to ipynb --pipe-fmt ipynb \
--pipe 'jupyter nbconvert --to ipynb --execute --allow-errors --stdin --stdout' \
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great example!

script.sh
```

In each of the above, `jupyter nbconvert` could be replaced with any alternative tool to execute a jupyter notebook non-interactively, including [papermill](https://github.com/nteract/papermill) which would allow notebook parameterisation.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my experience, papermill is a great tool, and by default it catches execution errors. May I ask if you have reasons not to use it in your personal workflow?

Also would you mind to add a link to a post I authored a few years ago on Jupytext + Papermill ? It is available on medium (
https://medium.com/capital-fund-management/automated-reports-with-jupyter-notebooks-using-jupytext-and-papermill-619e60c37330 ) and on github ( https://github.com/CFMTech/jupytext_papermill_post ), maybe adding the two links is important as medium looks nicer but github is more freely available...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RE papermill: this is why

$ jupytext --to ipynb --set-kernel bash test.sh --pipe-fmt ipynb --pipe 'papermill -f param.yml'
[jupytext] Reading test.sh in format sh
[jupytext] Setting kernel bash
[jupytext] Updating notebook metadata with '{"kernelspec": {"name": "bash", "language": "bash", "display_name": "Bash"}}'
[jupytext] Executing papermill -f param.yml
Traceback (most recent call last):
  File "/home/kevin/.local/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/usr/lib/python3/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/kevin/.local/lib/python3.9/site-packages/papermill/cli.py", line 250, in papermill
    execute_notebook(
  File "/home/kevin/.local/lib/python3.9/site-packages/papermill/execute.py", line 94, in execute_notebook
    nb = parameterize_notebook(
  File "/home/kevin/.local/lib/python3.9/site-packages/papermill/parameterize.py", line 83, in parameterize_notebook
    param_content = translate_parameters(kernel_name, language, parameters, comment)
  File "/home/kevin/.local/lib/python3.9/site-packages/papermill/translators.py", line 560, in translate_parameters
    return papermill_translators.find_translator(kernel_name, language).codify(parameters, comment)
  File "/home/kevin/.local/lib/python3.9/site-packages/papermill/translators.py", line 31, in find_translator
    raise PapermillException(
papermill.exceptions.PapermillException: No parameter translator functions specified for kernel 'bash' or language 'bash'
[jupytext] Error: The command 'papermill -f param.yml' exited with code 1 (use --warn-only to turn this error into a warning)

Though I really ought just make a PR there for Bash as it would be pretty simple.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As expected it turned out to be pretty easy. See nteract/papermill#674



## Notebook and cell metadata filters

If you want to preserve (or filter out) certain notebook or cell metadata, change the value of either `notebook_metadata_filter` or `cell_metadata_filter` with the `--update-metadata` option. For instance, if you wish to convert an `.ipynb` document to a `.md` file and preserve all the notebook metadata in that document, run
Expand Down