Meta extension: Empty line in YAML is considered as End of Document

[alixedi]

For:

PyYAML==3.11

This fails:

doc = """---
First name: John
Last name: Doe

email: [email protected]

---

Hello World!
"""

import markdown
md = markdown.Markdown(extensions=[markdown.extensions.meta.MetaExtension(yaml=True)])
assert md.convert(doc) == '<p>Hello World</p>'

I understand that YAML can have blank lines.

[alixedi]

Looking at meta.py, line 80, it appears this is deliberate?

if line.strip() == '' or have_yaml and YAML_END_RE.match(line):
    break # blank line or end of YAML header - done

[waylan]

I am very much inclined to reject this. I was hesitant to add support for YAML to begin with. Previously, we didn't even support the YAML style deliminators. At that time, a blank line was the only way to determine the end of the meta data and the start of the document. As you note, the implementation is still very much dependent on that blank line. And i intend to continue to support meta-data without YAML style deliminators for the foreseeable future. However, as stated in the documentation, this feature (meta-data) was inspired by MultiMarkdown which also includes support for YAML style deliminators (but not YAML data). Therefore, in the last release, we added support for the deliminators. However, it is the blank line that really ends the data section (the deliminators are just ignored as long as they are on the first or last line). Yes, this means that a blank line gets precedence over the closing deliminator in determining the end of the data section. This was an intentional design decision which gives us full conformity with MultiMarkdown's implementation.

In the process of adding the above mentioned support, someone requested that we also support YAML data. I was hesitant, but saw the value in users being able to have more control over the structure of their data so I included it as an optional feature. That said, I am more inclined to remove it that to expand it. It seems to me that if you want full-featured YAML support, a third party implementation would be better suited to your needs.

[facelessuser]

I kinda feel like YAML support could be its own plugin...maybe even should. When people see YAML support, they are going to think they can use the traditional YAML frontmatter (which is what they really want). But the truth is you can't, which I think is fine.

I kind of thought YAML was already implemented as 3rd party extension listed in the wiki https://github.com/teoric/python-markdown-yaml-meta-data, so I was kind of surprised to see it added officially. I haven't used the 3rd party or the official because I strip out YAML via a wrapper around Markdown to use as the data as actual HTML meta data, or use special keywords to set settings or dynamically control which extensions get loaded for a specific file.

[waylan]

@facelessuser in retrospect I agree. I suspect I will be removing the support for YAML data. I would have never implemented it myself, but when the pull-request came in with it already implemented, I accepted it without thinking it through fully. At the time, it didn't occur to me that YAML permits blank lines and that this would not offer full support. Taking a step back, it is now clear to me that the existing meta-data extension and YAML support are two very different things which should not exist together in the same extension.

I haven't used the 3rd party or the official because I strip out YAML via a wrapper around Markdown to use as the data as actual HTML meta data, or use special keywords to set settings or dynamically control which extensions get loaded for a specific file.

Personally, this is how I think it should be done. I'm always surprised how many people want to use the extension and continue to insist when I suggest they use a wrapper instead. I suppose it is because no one wants to reinvent the wheel. However, if someone wrote a good implementation which was available as a third party package (perhaps via PyPI), I might be inclined to deprecate the extension altogether.

As an aside, I wrote the original meta-data extension many-many years ago as an experiment. At the time I was a relatively new user of Python-Markdown only. However, when I announced the meta-data extension (which I believe was the third I had written) it caught the attention of the maintainer of Python-Markdown (Yuri at the time) and he asked me to join the project. At the time, no extensions were not being shipped with Python-Markdown and when we decided to start doing that, a number of my extensions were all added in at once. I was never particularly happy that meta-data was included because once i built it and started using it, I realized that a wrapper would make for a much more useful tool. I just never developed the need or found the time to write it myself.

[waylan]

Perhaps I should explain why I think this would be better as a wrapper than an extension. Consider two projects which make heavy use of the existing Meta-Data extension. mkdocs and Pelican. Both of those projects have a list of default extensions that their users will get out-of-the-box without changing any settings. In each instance, Meta-Data is one of those extensions (and it is required for things to work in each of those tools). However, each project offers a setting to their users where the user can define their own list of extensions. Naturally, that setting can be used to remove/replace the default list. However, what happens when the user removes the meta-data extension? Then the list needs to be checked to ensure that that extension is included and if not, insert it before passing the list on to Markdown. Otherwise things will break. Wouldn't it make more sense for those projects to just run the input file through some code which extracted the meta-data first, then pass the rest of the document on to Markdown without needing to juggle the extension list. And remember, the list of extensions can be class instances or string names. You can't just do a simple if not 'meta' in extensions. You need to loop through the list and check the type of each item. It is true that mkdocs is simpler as their settings file is YAML (however PyYAML does allow python objects to be specified in YAML files so I wouldn't rule the possibility out), but Pelican uses a Python file for settings. You can totally import an extension and create an instance to add to the list of extensions (which is great for the user to configure each extension how s\he desires). The point is, I would much rather see these projects doing something like this:

data, md = get_data(source)
html = markdown.markdown(md, **get_md_kwargs(config, data))
Template(body=html, context=get_context(config, data))

While that is mostly pseudo code which leaves the implementation details out, I think the general concept is a much better way of working with meta data. As an alternate example, the above functions could all be methods on a class. The important point is that the meta-data is extracted before passing the document on to Markdown.

[facelessuser]

With stuff like PyYAML, getting the frontmatter and parsing it is trivial. Something like this can be added to a wrapper real easy. Heck you could use whatever content you wanted, JSON, YAML, .

import re
import yaml


def get_frontmatter(string):
    """ Get frontmatter from string """
    frontmatter = {}

    if string.startswith("---"):
        m = re.search(r'^(---(.*?)---[ \t]*\r?\n)', string, re.DOTALL)
        if m:
            try:
                frontmatter = yaml.load(m.group(2))
            except:
                pass
            string = string[m.end(1):]

    return frontmatter, string

I struggled to find a reason that required Python Markdown to parse my frontmatter, when I realized there wasn't really any reason to have Python Markdown do it for me (as it did absolutely nothing with it once it stripped it out), I realized I should just parse it out myself and treat it exactly as it was, just extra data that Python Markdown doesn't care about. It worked out better this way because quite often, I wanted to do something with the info before I handed the actual text to Python Markdown.

[waylan]

Right. It occurs to me that when I first created the meta-data extension, JSON and YAML were, at most, in their very early stages (definitely not popular or standard library material) and so the benefit of having a parser of a key/value data format which was easy to write and required no additional dependencies was very attractive to users (the only standard library option was INI files). Today, given the great strides that have been made in python package management, third party dependencies are mostly a non-issue and the more fully developed data formats should be more desirable (PYAML's c dependency is optional so that's not a blocker either). In fact, the maintainers of those projects are better equipped to maintain that sort of thing than a Markdown project would be. That being the case, I think maybe meta-data will be deprecated in the near future. We should leave data parsing to data parsers.

[waylan]

To be clear, if I deprecate meta-data, the extension would be broken out into its own package which would get no further development by me. If other's wanted to use it or fork it, they certainly could, although, as stated above, there are better options.

[waylan]

@tomchristie or @d0ugal (from mkdocs) and @getpelican (from Pelican), do you guys have any input on the future of Python-Markdown's Meta-Data extension. I'd like to hear from you (presumably Python-Markdown's biggest users now that Django removed the markup contrib app) before doing anything drastic like deprecating an extension that both projects rely heavily on. See the earlier comments in this discussion for the details, but the current idea is to remove the broken YAML support immediately (in 2.6.1 as a bug-fix) as it has not even been a week and then start the deprecation process for the entire extension with the next major release which would be months out at the earliest.

Oh and look for a plan for a 3.0 published sometime in the near future. I'm ruminating on it now and will start to put something together soon. I don't expect much more in the 2.x series (3.0 might be next or if not, after 2.7).

[kernc]

Could something like

if not have_yaml and line.strip() == '' or have_yaml and YAML_END_RE.match(line):

be used for removingfixing broken YAML support?

[waylan]

@kernc that could work, but I'm not sure we should. One thing is certain, I'm not interested in maintaining YAML support long-term. So every time I get a bug report from here on out, I'll wish I had ripped it out now. I'm not sure why people are so insistent that the built-in extensions must support their desired feature. Why can't this be in a third party extension? Especially now that the built-in extensions get no preferential treatment.

[tomchristie]

@waylan I've no problem with it being moved into a third party package - certainly sounds like the right approach for your project, and it's probably not much work for us - if you wanted to handle opening a ticket on MkDocs pointing us at the right place etc for the new package, then I can't see any objection.

[d0ugal]

+1, I agree with @tomchristie.

[waylan]

For anyone who is interested, I threw together a package that parses meta-data as I would want it to work. It is a standalone package (not an extension) and it can work with any lightweight markup language (not just Markdown). I haven't written the YAML implementation yet, but the MultiMarkdown style meta-data is feature complete. You can even define types for specific keys. Any feedback is welcome (please send feedback through that project not here). Thanks.

https://github.com/waylan/docdata