Black has syntax error with valid iPython magic when using --pipe

[Skylion007]

To reproduce:

This occurs on black, version 19.10b0 and 'jupytext==1.5.2'. When --sync a notebook to py:percent format using the following command:

jupytext --update-metadata '{"jupytext":{"notebook_metadata_filter":"all", "cell_metadata_filter":"-all"}, "accelerator":"GPU"}' --set-formats 'nb_python//py:percent,colabs//ipynb' --pipe black --pipe "sed s/[[:space:]]*\#[[:space:]]\%\%/\#\%\%/g"  --pipe 'isort -' --pipe-fmt 'py:percent' --sync

I get this error:

executing black -
error: cannot format -: Cannot parse: 73:0: !~/miniconda.sh -b -p $HOME/miniconda

Now I tracked down the error, and when I change the lines ipynb to use !bash ~/miniconda.sh -b -p $HOME/miniconda, it works just fine. Also the following lines also generated error:

error: cannot format -: Cannot parse: 78:0: !. activate habitat;``` 

I had to change it to
```bash
!conda . activate

I also had to change:

!./test_locally_pointnav_rgbd.sh

to

!bash ./test_locally_pointnav_rgbd.sh

TLDR: Black has issues when the first arg following '!' isn't a command, but references an executable file or another bash alias.

[Skylion007]

One other point is that the entire generated script appears to be commented (no non-ipython magic) so black was probably trying to parse it as a shebang!

[mwouts]

Pretty odd behavior, maybe the ipython magic should be commented out before handing it over to the Black formatter? After all, it's perfectly valid to run these cells in colab.

Oh I see... the regular expression at

jupytext/jupytext/magics.py

Line 43 in 213acee

_PYTHON_HELP_OR_BASH_CMD = re.compile(r"^\s*(# |#)*\s*(\?|!)\s*[A-Za-z]")

is a bit too restrictive - you want to allow ~ or . after the exclamation mark, is that correct?

Would you like to propose a PR with

• a fix on the _PYTHON_HELP_OR_BASH_CMD regular expression,
• together with two more examples of magic commands at
  

  jupytext/tests/test_escape_magics.py

  Lines 157 to 175 in 213acee

  	@pytest.mark.parametrize(
  	"magic_cmd",
  	[
  	"ls",
  	"!ls",
  	"ls -al",
  	"!whoami",
  	"# ls",
  	"# mv a b",
  	"! mkdir tmp",
  	"cat",
  	"cat ",
  	"cat hello.txt",
  	"cat --option=value hello.txt",
  	],
  	)
  	def test_comment_bash_commands_in_python(magic_cmd):
  	assert comment_magic([magic_cmd]) == ["# " + magic_cmd]
  	assert uncomment_magic(["# " + magic_cmd]) == [magic_cmd]

  
  inpired by the ones you report above?

One other point is that the entire generated script appears to be commented (no non-ipython magic) so black was probably trying to parse it as a shebang!

I am not sure to follow, can you give a MRE? Thanks!

[Skylion007]

This diff has the faulty ipynb in it: facebookresearch/habitat-lab@a96e0f4

Archived in case that diff get's squashed
https://web.archive.org/web/20200819164106if_/https://github.com/facebookresearch/habitat-lab/pull/455/commits/a96e0f4ede2d7df3df47a2b3991a795b4aa3cd5f

This is the diff that fixed it: facebookresearch/habitat-lab@2ff9416

[Skylion007]

@mwouts I opened a PR that solves the issue.

[mwouts]

Thanks @Skylion007 for the fix. And for the notebook examples. Great project by the way!