The logs in jupyter are too verbose + notebook is not trusted after black / reload

[mwouts]

Here is an short extract of the logs of my jupyter lab:

[I 2021-05-30 22:03:35.450 ServerApp] Saving file at /Fibonacci.py
[I 2021-05-30 22:03:35.451 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:03:35.455 ServerApp] Saving Fibonacci.ipynb
[I 2021-05-30 22:03:35.509 ServerApp] Saving Fibonacci.py in format py:percent
[I 2021-05-30 22:03:36.338 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:03:36.364 ServerApp] Saving file at /Fibonacci.py
[I 2021-05-30 22:03:36.365 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:03:36.369 ServerApp] Saving Fibonacci.ipynb
[I 2021-05-30 22:03:36.437 ServerApp] Saving Fibonacci.py in format py:percent
[I 2021-05-30 22:03:46.521 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:03:51.393 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:03:51.399 ServerApp] Reading OUTPUTS from Fibonacci.ipynb
[W 2021-05-30 22:03:51.401 ServerApp] Notebook Fibonacci.ipynb is not trusted
[I 2021-05-30 22:03:53.760 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:03:53.768 ServerApp] Saving file at /Fibonacci.py
[I 2021-05-30 22:03:53.768 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:03:53.769 ServerApp] Saving Fibonacci.ipynb
[W 2021-05-30 22:03:53.769 ServerApp] Notebook Fibonacci.ipynb is not trusted
[I 2021-05-30 22:03:53.779 ServerApp] Saving Fibonacci.py in format py:percent
[I 2021-05-30 22:04:03.838 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:04:13.878 ServerApp] Loading Jupytext configuration file at /.jupytext.toml

I would prefer to have

• less mentions of "Loading Jupytext configuration file at /.jupytext.toml"
• less redundancy in logs when paired files are saved: in the below we have two lines /Fibonacci.py and Fibonacci.py but there is a single .py file

[I 2021-05-30 22:03:36.364 ServerApp] Saving file at /Fibonacci.py
[I 2021-05-30 22:03:36.365 ServerApp] Loading Jupytext configuration file at /.jupytext.toml
[I 2021-05-30 22:03:36.369 ServerApp] Saving Fibonacci.ipynb
[I 2021-05-30 22:03:36.437 ServerApp] Saving Fibonacci.py in format py:percent

• And I'd prefer not to have this warning Notebook Fibonacci.ipynb is not trusted, that came after I ran black and reloaded the .py notebook.

[mwouts]

In a debug session with Jupytext + Jupyter I was seeing missing cell ids when the notebook is saved, and these missing ids were the cause for trust issues (since the notebook was signed without the cell ids)

In another session with just JupyterLab I can't reproduce this... On the Jupytext session I had these versions

jupyter core     : 4.7.1
jupyter-notebook : 6.2.0
qtconsole        : not installed
ipython          : 7.20.0
ipykernel        : 5.3.4
jupyter client   : 6.1.11
jupyter lab      : 3.0.8
nbconvert        : 6.0.7
ipywidgets       : not installed
nbformat         : 5.1.3
traitlets        : 5.0.5

while on the Jupyter-only session I have

jupyter core     : 4.7.1
jupyter-notebook : 6.4.0
qtconsole        : not installed
ipython          : 7.25.0
ipykernel        : 6.0.1
jupyter client   : 6.1.12
jupyter lab      : 3.0.16
nbconvert        : 6.1.0
ipywidgets       : not installed
nbformat         : 5.1.3
traitlets        : 5.0.5

So it seems that the missing cell ids were fixed in between.

NB: Apparently the cell ids have been added to Jupyter Lab 3.0.13 (jupyterlab/jupyterlab#10018)

[weiji14]

Hi there, first off, thanks for you wonderful tool! Jupytext has become one of my must-have dependencies in all my recent projects 😁

Anyways, just noticed the same verbose result too on jupytext 1.12.0 with configuration in pyproject.toml. Seems to be printing a line every 10 seconds! Is there a way to make it print to the console less often? Maybe changing the level from info to debug or something?

[I 2021-09-14 17:40:56.952 ServerApp] Loading Jupytext configuration file at /pyproject.toml
[I 2021-09-14 17:41:07.361 ServerApp] Loading Jupytext configuration file at /pyproject.toml
[I 2021-09-14 17:41:17.651 ServerApp] Loading Jupytext configuration file at /pyproject.toml
[I 2021-09-14 17:41:28.062 ServerApp] Loading Jupytext configuration file at /pyproject.toml
[I 2021-09-14 17:41:38.469 ServerApp] Loading Jupytext configuration file at /pyproject.toml

Relevant installed dependencies:

jupyter-client            7.0.2
jupyter-core              4.7.1
jupyter-server            1.10.2
jupyterlab                3.1.10
jupyterlab-pygments       0.1.2
jupyterlab-server         2.7.2
jupytext                  1.12.0
nbclassic                 0.3.1
nbclient                  0.5.4
nbconvert                 6.1.0
nbformat                  5.1.3

[mwouts]

Thank you @weiji14! Well currently there is a line every time the configuration file is reloaded:

jupytext/jupytext/contentsmanager.py

Lines 487 to 491 in e1c4696

	def load_config_file(self, config_file, is_os_path=False):
	"""Load the configuration file"""
	if config_file is None:
	return None
	self.log.info("Loading Jupytext configuration file at %s", config_file)

And the config file is reloaded every time a notebook/file/folder is opened/created or saved, and also even when the content of a directory is listed (although in that last case we use a one-second cache).

I think it is helpful to let the user know which config file is being used, and also I prefer to have the log every time the config is actually read.

So maybe I would prefer to only change the behavior for folders, i.e. increase the duration of the cache. What do you think? Would you like to experiment with different values and let us know which duration you find best in the below?

jupytext/jupytext/contentsmanager.py

Line 510 in e1c4696

self.cached_config.timestamp + timedelta(seconds=1) < datetime.now()

[weiji14]

Hmm, a 1 second cache seems too fast... I doubt configuration files change very often, maybe increase it to something like 5 minutes? I'm still thinking whether this could be logged at the debug-level instead. Or, if you prefer keeping it at the info-level, only print out the log message if the configuration file did change from last time (e.g. change the message from "Loading Jupytext config..." to "Loaded Jupytext config...").

[mwouts]

Hi @weiji14 , a quick comment on the resolution for this issue... I decided to refresh the config only when a notebook is opened, saved, or when a new directory is listed - No more periodical updates, so much less logs. The difference with the previous behavior is only for listing directories, so I do not expect much impact on users (the updated config will be used anyway when they open a file).