Rich Text editing on the site

[nimmolo]

Switching to Markdown

@JoeCohen is researching switching from Redcloth gem to Redcarpet (which implies a switch from Textile to Markdown) and reports:

At the moment I'm trying to figure out how the redcarpet gem works and understand how we use Redcloth so that we can use redcarpet as a substitute for Redcloth, which is no longer maintained and has a security warning. (I'm having trouble getting my head around how to do this. Maybe I somehow have to trick redcarpet into interpreting MO-flavored Textile as "standard" Markdown. Then redcarpet will correctly render the result as html.

If I can do that, then we should be able to render standard markdown as html, see example here, and maybe even extend redcarpet so that we can render Github flavored markdown as html.)

Textile and Markdown are two similar markup languages, but they're different. After looking Redcarpet over I don't believe it can convert Textile on the fly. Maybe it can be tricked into doing so, but that seems unwieldy.

If there's consensus it's advisable to switch from Textile to Markdown, I imagine we would need to do database conversions (migrations) of existing Textile into Markdown using something like the redmine reformat gem. I believe it's a chunk of work, but I'm in favor. However, as far as I can tell, these db conversions would NOT need to happen all at once, and should probably be done field by field (see outline below). Here's how i picture the steps involved:

Please edit directly as new discoveries come to light

• Our view templates currently take Textile strings from the db and convert them to HTML using MO's String extensions like t and tpl. These just call methods of the Textile class to generate HTML. So we'd want to write a new class that does Markdown conversions to HTML, analogous to our Textile class. It should do everything our Textile class does, pretty much method for method.
• Textile invokes Redcloth rendering methods, but it also does a lot of special handling of Names. Thankfully, I believe there is no difference between Textile and Markdown regarding the bold/italic markup used in MO's Names (is this true?), so names themselves would not need to undergo db conversions.
• We'd want to write parallel string extensions that call the methods in this new Markdown class, analogous to our extensions like tpl that call Textile methods.
• Then try db conversions, one field at a time. Once the text in, say, Observation Notes is converted to Markdown, we'd rephrase the view erb to use our new markdown string extensions and Markdown class methods to render the text from the database, instead of the Textile class we use now.

Here's the current erb, with notes on what it's doing

  <%= if observation.notes?
    Textile.clear_textile_cache  # clears class variables relating to looking up names
    Textile.register_name(observation.name)  # instantiates a name object
    # 'tpl' is a method of StringExtensions that calls methods of our Textile class
    content_tag(
      :div, observation.notes_show_formatted.sub(/^\A/, "#{:NOTES.t}:\n").tpl
    )
  end %>

tpl calls Textile.textilize, the core of which is:

red = RedCloth.new(self)

An analogous string method that calls Redcarpet::Markdown class might be something like this (from the Redcarpet docs):

markdown = Redcarpet::Markdown.new(renderer, extensions = {})
markdown.render(self)

• The gems could live alongside each other during the conversion, and templates that handle the data could go field by field, view by view or controller by controller. I can't remember how often we use textile, but it's a lot of places.

Making info pages editable

Currently our Info pages are assembled from long translation strings that use Textile.

It could be nice to have a standard UI for editing page content for any page in the Info controller and storing that content in the DB by storing reference to the translation string for the page ID, more like a CMS.

[JoeCohen]

I was thinking of trying to modify our current Textile to:

• use redcarpet parser instead of Redcloth
• on-the-fly convert Texttile to redcarpet flavored markdown when the two are different
• use redcarpet renderder to turn everything into html.

[nimmolo]

I see. That seems harder to pull off, to me.

• If you are changing the Textile methods in one sweep, everything has to be 💯 the first time, little room for unforeseen, untested errors
• In the event of an edit, what would you write back to the db, after parsing the Textile — Textile or Markdown?
• If you're parsing Textile as Markdown for the view but resaving as Textile... that sounds complicated
• If the db for a particular field may have either, even the template parsing seems like it would be quite complicated

[JoeCohen]

• I didn't understand about writing back to the db. I'd just leave the db alone.
• I don't understand why doing it on the fly would be more difficult than converting the db to Markdown.
  Note that we currently do some on the fly, notably all the _blah_ constructs.
  I'm thinking that all (or almost all) can be handled in the current Textile class by a series of gsub!s. E.g. gsub!(".h6", "#####")
• A hard problem (at least for me) is dealing with inlines and fenced code. We currently don't do this. But there are work-arounds. See https://www.pivotaltracker.com/story/show/185718141
• I will try to discuss this at the next Tech. Meeting.

[nimmolo]

Sounds like a good subject for the meeting.

My reference to the db is just this:
Say you have a Notes field in Textile (for an observation, or a herbarium, or whatever record):

• the string is stored in the db as a string with Textile markup
• When it's displayed on the site, it's parsed in the template by our Textile class and displayed as HTML
• When someone edits the record, the Notes field is pulled from the db into the textarea unaltered, with all its Textile markup
• When they save the update, the altered Textile markup is stored back into the db.

I guess i'm picturing a scenario where we gradually switch to having users use Markdown for fields like that.
In that case, converting the db entry for a field from Textile marks to Markdown marks seems like the elegant solution, and then users can edit the Markdown, and we can set the template to parse the field as Markdown, no hacks involved.

For the transition period, some user-editable fields would be Textile while others would be Markdown. But we already have links to a Textile cheat sheet wherever a field is textile (i would like to make this a popup modal), so we could just link to Markdown cheatsheet where it's markdown, until the conversions are done.

Although this process is initially more laborious and requires a bit more attention from users, to me it seems more confusing to have to maintain a system where users edit in Textile but we parse the Textile with a hacked Markdown parser... no?

[JoeCohen]

I put a bunch of links in Pivotal, including to more-or-less official standards.
https://www.pivotaltracker.com/story/show/185627719/comments/237787512