Add Markdown YAML header fields to notebook metadata

[dlukes]

I would like to add custom fields to the Markdown YAML header (title: and author:) and have them end up in the notebook's "metadata": field upon conversion from Markdown to .ipynb, instead of being copied as literal YAML into a raw cell at the beginning of the notebook. Is there a way to achieve that?

I tried the following but it doesn't seem to work:

jupytext --to notebook test.md --update-metadata '{"jupytext": {"notebook_metadata_filter":"title,author"}}'

The title: and author: YAML fields stay in the raw cell, and the notebook's "metadata": field contains just literally what I passed to the --update-metadata option, i.e. {"jupytext": {"notebook_metadata_filter":"title,author"}}.

[dlukes]

Alternatively, is there a way to specify this metadata in some other way than in the YAML header, so that it ends up stored in the notebook-wide "metadata": field? :)

[dlukes]

Here's what that input markdown test.md file could look like:

---
author: John Doe
title: Some serious math
jupyter:
  jupytext:
    text_representation:
      extension: .md
      format_name: markdown
      format_version: '1.2'
      jupytext_version: 1.4.1
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---

This is the output notebook file that I get by running the command in OP:

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "---\n",
    "author: John Doe\n",
    "title: Some serious math\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "notebook_metadata_filter": "title,author"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

And this is what I'm trying to achieve:

{
 "cells": [],
 "metadata": {
  "author": "John Doe",
  "title": "Some serious math",
  "jupytext": {
   "notebook_metadata_filter": "title,author"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

[mwouts]

Hello @dlukes , thanks for asking! You were very close to finding the solution... jupytext --to md on your last example would have yield the answer, e.g.

---
jupyter:
  author: John Doe
  jupytext:
    notebook_metadata_filter: title,author
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
  title: Some serious math
---

As you notice, the notebook metadata is always under the jupyter section, at least in the default md format (in the new md:myst format, the notebook metadata is at the root of the YAML header). If you start from the markdown file, Jupytext should take care of creating/updating the metadata filter, so that you don't lose metadata that you have inserted manually.

With the PR #471 we can add an example of this in the tests, but maybe you would have found useful to have a mention of this in the documentation... Where do you think we should add this? Thanks

[dlukes]

D'oh! So close :) On the other hand, if this information makes it into the docs, I hope it will have been worth it, I think it will be useful for other people who use Jupytext with Jupyter Book.

Where do you think we should add this?

I searched the docs for 'metadata', so I looked here (FYI, that section has a broken link to here which should probably point here instead?) and here.

However, I think it would be most natural to put it in the Jupytext Markdown format description. There's already an example YAML block there, so add the custom metadata to that and one or two sentences commenting on it in the main text?

And thank you for Jupytext as well as for responding so quickly!

[mwouts]

I have added a quick note about custom notebook metadata as you suggested it (and fixed the broken link). Thanks for your getting in touch!

[dlukes]

Thanks for addressing this so quickly!