How do I keep 2 lines after functions when piping black?

[benlindsay]

I'm loving jupytext and the ability to keep my code clean and lint-free! I'm just trying to get my workflow as clean as possible and running into one little snag. flake8 wants to have 2 empty lines after functions, but when piping black via jupytext, those 2 lines don't get preserved for me.

How to replicate:

1. Create a file called notebook.py with the following content:

   # %%
   def func():
       return 42
   # %%
   func()

2. Create a synced .ipynb file:

   $ jupytext --set-formats ipynb,py:percent --sync notebook.py
   [jupytext] Reading notebook.py in format py
   [jupytext] Updating notebook metadata with '{"jupytext": {"formats": "ipynb,py:percent"}}'
   [jupytext] Executing black -
   All done! ✨ 🍰 ✨
   1 file left unchanged.
   [jupytext] Updating notebook.ipynb
   [jupytext] Updating notebook.py
   $ cat notebook.py
   # ---
   # jupyter:
   #   jupytext:
   #     cell_metadata_filter: -all
   #     formats: ipynb,py:percent
   #     text_representation:
   #       extension: .py
   #       format_name: percent
   #       format_version: '1.3'
   #       jupytext_version: 1.10.3
   # ---
   
   # %%
   def func():
       return 42
   # %%
   func()
   $ cat notebook.ipynb
   {
    "cells": [
     {
      "cell_type": "code",
      "execution_count": null,
      "id": "8e0e0c0c",
      "metadata": {
       "lines_to_next_cell": 0
      },
      "outputs": [],
      "source": [
       "def func():\n",
       "    return 42"
      ]
     },
     {
      "cell_type": "code",
      "execution_count": null,
      "id": "745567b8",
      "metadata": {},
      "outputs": [],
      "source": [
       "func()"
      ]
     }
    ],
    "metadata": {
     "jupytext": {
      "cell_metadata_filter": "-all",
      "formats": "ipynb,py:percent",
      "main_language": "python"
     }
    },
    "nbformat": 4,
    "nbformat_minor": 5
   }

3. Lint file to realize that flake8 wants 2 spaces after functions:

   $ flake8 notebook.py
   notebook.py:17:1: E305 expected 2 blank lines after class or function definition, found 0
   

4. Run black on the .py file to add the 2 spaces back in, satisfying flake8:

   $ black notebook.py
   reformatted notebook.py
   All done! ✨ 🍰 ✨
   1 file reformatted.
   $ flake8 notebook.py
   $
   

5. Pipe black through jupytext, which gets rid of those 2 lines again:

   $ jupytext --sync --pipe black notebook.py
   [jupytext] Reading notebook.py in format py
   [jupytext] Loading notebook.ipynb
   [jupytext] Executing black -
   reformatted -
   All done! ✨ 🍰 ✨
   1 file reformatted.
   [jupytext] Updating notebook.py
   $ flake8 notebook.py
   notebook.py:17:1: E305 expected 2 blank lines after class or function definition, found 0
   

6. Manually change the lines_to_next_cell metadata from 0 to 2 in the notebook.ipynb, then repeat:

   $ jupytext --sync --pipe black notebook.py
   [jupytext] Reading notebook.py in format py
   [jupytext] Loading notebook.ipynb
   [jupytext] Executing black -
   All done! ✨ 🍰 ✨
   1 file left unchanged.
   [jupytext] Updating notebook.py
   $ flake8 notebook.py
   $
   

It seems to me that the problem is that the lines_to_next_cell metadata is sticky, and changing the lines between cells in the python file doesn't affect that value after the cell is initially created. Is that something that could change? Could you make it so that when the .py file is newer, the lines_to_next_cell value gets updated in the notebook if there are changes, or would that break other functionality? And if that would break things, is there another solution for this kind of situation besides turning off that particular error in flake8, which will be my current solution?

[mwouts]

Hi @benlindsay , thanks for reporting this, yes I agree when piping through black or similar tools we don't want to retain the lines_to_next_cell information (that information is used in order to provide round trip stability when doing py -> ipynb -> py, but is this context the round trip identity is not what we want...)

I'll have a look at this and hopefully propose a fix in the coming days.

[mwouts]

Hi @benlindsay , I think I have reproduced (and fixed) the issue at #775 .
I hope to ship a new release by the end of the week-end.
Thanks again for reporting it!

[benlindsay]

Thanks @mwouts for working on this and the fast turnaround time! I'll give this a shot next time I get a chance!

[mwouts]

You're welcome! Sure I'd be happy to know how the new version (now on pypi) works for you. Please let me know if it works as expected, or feel free to reopen if not!