Catch errors when running --pipe black

[psychemedia]

If you try to run the --pipe black command over a set of notebooks in one or more directories, and black fails for some reason, the pipe fails at that point.

Non-runnable illustrative fragment:

jupytext --sync  --pipe black  Part*/*.ipynb

---

[jupytext] Executing black -
reformatted -
All done! ✨ 🍰 ✨
1 file reformatted.
[jupytext] Updating 'Part 03 Notebooks/03.2 Selecting and projecting, sorting and limiting.ipynb'
[jupytext] Updating 'Part 03 Notebooks/.md/03.2 Selecting and projecting, sorting and limiting.md'
[jupytext] Reading Part 03 Notebooks/03.3 Combining data from multiple datasets.ipynb in format ipynb
[jupytext] Loading 'Part 03 Notebooks/.md/03.3 Combining data from multiple datasets.md'
[jupytext] Executing black -
error: cannot format -: Cannot parse: 156:9: result = %sql SELECT * FROM quickdemo WHERE value > 25
Oh no! 💥 💔 💥
1 file failed to reformat.

In the above case, black failed trying to parse a line that assigned the result of a line magic call to a variable.

It would be useful if the Jupytext pipe did not fail, but did perhaps warn that a file could not be parsed.

Another solution might be to allow a cell to be tagged as no-black or similar and be ignored somehow (--ignore-tag no-black?) (I'm not sure if you apply the formatter at the cell level?)

[psychemedia]

Ah, I see that black maybe supports exclusions which can be explicitly include in code:

# fmt: off
myvar = %dont_format_me
# fmt: on

[via]

BUT: hmm... no, still get the error.

[mwouts]

Hi @psychemedia , sorry for the late answer.
Well I am not always sure about catching errors, but maybe yes we could add a --catch-errors mode?

Also more specifically for the example above, did you try adding an active-ipynb tag to the cell that contains the problematic instruction?

[psychemedia]

Ah... good tip.. active-ipynb; so the black formatter ignores that? (DO other --pipe elements also skip those cells?)

[mwouts]

Yes, cells with an active-ipynb tag will be commented out in scripts, so they should not cause issues in black. However if you want to run flake8 then it will be a bit different, as you might need to provide an alternative cell to make sure that the variable myvar is defined even when the problematic cell is commented out.

Also you're seeing this because Jupytext does not identify the line myvar = %dont_format_me as a magic command, maybe we could improve that part as well (magic commands are always commented out by default).

[psychemedia]

The detections of magics in formatters and linters is a general issue I think, with folk divided as to how that sort of exception should be handled.

[mwouts]

Hi @psychemedia , I expect to have fixed both issues in the development version. Feel free to give it a try, cf. https://jupytext.readthedocs.io/en/latest/developing.html

[MarcoGorelli]

Jupytext does not identify the line myvar = %dont_format_me as a magic command, maybe we could improve that part as well

Suggestion for how to do this:

>>> from IPython.core.inputtransformer2 import TransformerManager
>>> TransformerManager().transform_cell('myvar = %dont_format_me')
"myvar = get_ipython().run_line_magic('dont_format_me', '')\n"

Then, using ast.parse, detect lines with get_ipython().run_line_magic (and other functions which magics would get transformed to) and comment them out (e.g. of what I'm referring to: psf/black#2357 , where jupytext has been mentioned)