[Request] Contribute utility function(s) for converting highlighted html to line-oriented html

[hallvard]

Is your request related to a specific problem you're having?
I needed to color individual lines based on the result of a diff, and this is difficult to do since span-elements may span several lines.

The solution you'd prefer / feature you'd like to see added...
I'd like a utility function for post-processing the highlighted html into a line-oriented html structure.

Any alternative solutions you considered...
Tried existing solution that failed in newest version. Also tried marking the html, but that's not allowed anymore.

Additional context...
Just to make it clear, I've implemented what I need and would like to contribute rather than host it myself (it's so small).

The lineify function below (with the help of lineifyHelper) replaces the highlighted html structure with a corresponding one, but where the top-level span-elements represents a single line of text/code. The previous contents has effectively been split up so the same kind of span-elements contain the same text/code, but the structure is line-oriented. This makes it easy to give specific lines a different look and feel (see the markLines argument) or add line numbers.

This code could be wrapped in a small module that can be used for post-processing. I'd like to avoid having to host this myself, therefore I'd like you to consider it.

function lineify(element, markLines) {
   if (typeof(element) == "string") {
      element = document.getElementById(element);
   }
   if (! element.classList.contains(highlightjsClass)) {
      element = element.querySelector(highlightjsSelector);
   }
   let lineNodes = [ createLineNode() ];
   lineifyHelper(element, [], lineNodes);
   // remove children
   while (element.firstChild) {
      element.removeChild(element.lastChild);
   }
   // add new line nodes
   for (let lineNode of lineNodes) {
      element.appendChild(lineNode);
   }
   // mark lines
   if (markLines) {
      for (let markLine of markLines) {
         lineNodes[markLine].classList.add("marked")
      }
   }
}

function lineifyHelper(element, elementStack, lineNodes) {
   let children = Array.from(element.childNodes)
   while (children.length > 0) {
      let child = children.shift();
      if (child.nodeType === Node.ELEMENT_NODE) {
         // element
         let childCopy = copyElementAndAttributes(child);
         appendChildCopyHelper(childCopy, elementStack, lineNodes);
         elementStack.push(childCopy)
         lineifyHelper(child, elementStack, lineNodes);
         elementStack.pop();
      } else if (child.nodeType === Node.TEXT_NODE) {
         // text
         let pos = child.data.indexOf("\n");
         if (pos < 0) {
             let childCopy = document.createTextNode(child.data);
             appendChildCopyHelper(childCopy, elementStack, lineNodes);
         } else {
            // append final text node
            let childCopy = document.createTextNode(child.data.substring(0, pos + 1));
            appendChildCopyHelper(childCopy, elementStack, lineNodes);
            // add new empty line node, that collects subsequent elements
            let lineNode = createLineNode();
            lineNodes.push(lineNode);
            // create new element stack representing start of new line
            for (let i = 0; i < elementStack.length; i++) {
               let elementCopy = copyElementAndAttributes(elementStack[i]);
               if (i === 0) {
                  lineNode.appendChild(elementCopy);
               } else {
                  elementStack[i - 1].appendChild(elementCopy)
               }
               elementStack[i] = elementCopy;
            }
            // enqueue rest of text node
            if (pos + 1 < child.data.length) {
               children.unshift(document.createTextNode(child.data.substring(pos + 1)));
            }
         }
      }
   }
}

function createLineNode() {
   let lineNode = document.createElement("span");
   lineNode.setAttribute("class", "code-line")
   return lineNode;
}

function appendChildCopyHelper(copy, elementStack, lineNodes) {
   let arr = (elementStack.length > 0 ? elementStack : lineNodes);
   arr[arr.length - 1].appendChild(copy);
}

function copyElementAndAttributes(element) {
   let copy = document.createElement(element.nodeName);
   for (var attr of element.attributes) {
      copy.setAttribute(attr.nodeName, attr.nodeValue);
   }
   return copy;
}

[taufik-nurrohman]

Have you tried the plugin API:

hljs.addPlugin({
  'after:highlightElement': ({ el, result }) => {
    el.innerHTML = lineify(el, [1, 4]);
  }
});

I'd like to avoid having to host this myself, therefore I'd like you to consider it.

But, the hljs.highlightAll() need to be written inline in your site. So you could just add hljs.addPlugin( ... ) after it, along with your code snippets. No need to host the snippet somewhere.

[hallvard]

The issue is not about helping me, I am using the code in my files by including calls to highlightjs before calling lineify, it's about contributing the code so others may use it. E.g. a utility module hosted @ highlightjs.

[taufik-nurrohman]

Ah, okay.

[joshgoebel]

I needed to color individual lines based on the result of a diff,

I'm sure you're aware we can already highlight diff or patch files - without any hoops if they are actually annotated as much...? But that's really a digression.

---

I'm not sure about random "utility" scripts (that's a very, very wide hole - really your code could be used with any HTML snippet, it's not specific to us) - though we could probably figure out a place where we might link to such utilities...

I'd be more interested in seeing if this couldn't be made into a proper plugin... and in that case I believe we'd be happy to host the repository I think - if you were willing to maintain it.

[hallvard]

This is a pretty custom solution used for reading exam submissions. The students start from a template and when reading their submission it's nice to see which lines they changed. A generated diff isn't suitable.

The code will technically work for "any" HTML, but the results won't be of practical use unless the HTML is limited to span-like elements, like what highlightjs generates. If highlightjs suddenly changes to a different generation strategy, it might stop working. That's why I think it makes sense to include it in a utility module.

It shouldn't be hard to turn it into a proper plugin, I agree. Do you have a plugin (repo) template, that I can clone and use as a starting point? That'll make it easier for both you and me.

[joshgoebel]

Nope, sadly, I don't think we have any official plugins yet. :) I'd suggest rollup for building both ESM and CJS bundles. You could look at https://github.com/highlightjs/vue-plugin just for the rollup building stuff, but it's not a plugin in the same sense as what I'm talking about. And I don't think you'd want us as a dependency (since that's already causing problems for the Vue plugin).

All you need to do is export your class or function that is a plugin so someone could call addPlugin and pass it.

Then it's worth discussing how someone could control it per element (which seems to be what you'd want to do here). I don't think that has received much thought in the past vs plugins of a more "global" nature - ie behaving the same everywhere. I'd suggest you embed the configuration in the HTML itself, but I'd be open to a discussion about the possibility of pass-thru of options when calling the API functions themselves. Though at that point it's worth asking whether a wrapper vs plugin might not be better.

I think we're exploring some new territory here, but I'm happy to help.

[hallvard]

I will look at rollup, at least I now have it working as a class with options, e.g. for prefixing each line with the line number. And you're right that using it as a plugin has issues. The general lineify-part works as a plugin, but the part that highlights specific lines needs to be configured per element, and that's a problem. I'll have to think about how to provide element-specific arguments.

[joshgoebel]

but the part that highlights specific lines needs to be configured per element, and that's a problem.

Right, that's what I was getting at. I see 3 ways forward there:

• wrap us highlightElementWithLineThingies(el, {...}), at which point you aren't technically using the plugin architecture
• use HTML <pre><code data-marklines="5,13,25"> for passing options
• we take options and pass them to plugins highlightElement(el, opts)

I imagine option 2 would actually work for a lot of use cases. Number one seems kludgy but I'd have to think thru the ramifications of the option 3. Right now highlightElement has no options to speak of at all.

[joshgoebel]

I'm a bit leaning towards #2... we don't currently allow someone to specify language (or ignoreIllegals, etc.) as options to highlightElement... it's rather assumed that everything needed to do the highlighting can be fetched from the element itself.

[hallvard]

I think I can solve my case by taking a callback argument in the plugin constructor, that will be called for each line in each code block. The code block would need some key attribute i.e. solution 2, so the callback can look up which lines should be styled or marked somehow. The callback could also add line numbers in whatever way it seems fit. I can also support some specific attributes in the code block, like you suggest, for common use cases (like adding line numbers).

[hallvard]

It worked pretty well as outlined. I added a code-block-key attribute and apply the callback to the highlighted element and the array of line elements (span). The callback may modify the line elements before they're added to the element. Hence, option 2 works for my use case.

[joshgoebel]

So what's it look like?

[hallvard]

Here's the callback:

function lineifyCallback(element, lineElements, diffLines) {
   let key = element.getAttribute("code-block-key");
   let lineNums = diffLines[key];
   for (let i = 0; i < lineElements.length; i++) {
      let lineElement = lineElements[i];
      lineElement.insertBefore(this.createLineNumSpan(i + 1), lineElement.firstChild);
   }
   for (let lineNum of lineNums) {
      lineElements[lineNum].classList.add("marked");
   }
}

And here's the code registering the plugin and running highlightjs:

		let diffLines = {
			VaccineTrial_java: [2,3,4,7,8,9,10,20,21,26,28,29,30,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,72,73,74,75,76,77,89,90,91,92,93], 
			VaccineTrialVolunteer_java: [3,4,5,8,9,13,18,24,25,26,34], 
			UniversityHandbook_java: [38,39,40,41,42,55,56,57,58,59], 
			UniversityHandbookUtils_java: [7,20,30,48,49,50,51,52,53,54], 
		};
		hljs.addPlugin(new LineifyPlugin(lineifyCallback, diffLines));
		hljs.highlightAll();

And here's a screenshot (left: submission with student additions to the template emphasised, right: junit test reports):

[hallvard]

https://github.com/hallvard/highlightjs-lineify

I've tried highlightjs-lineify.min.js file in dist instead of the original one in my project and it seems to work.

[joshgoebel]

What's stopping you from putting the data inside the HTML itself instead of just the key?

[hallvard]

Nothing, really. This way it was a smaller change to the page generator.
I've protyped logic for allowing any html attribute looking like xyzLines="2, 5, ..." to add xyz as class to corresponding line span elements. Then css can do the styling.

[joshgoebel]

 for (let i = 0; i < lineElements.length; i++) {
      let lineElement = lineElements[i];
      lineElement.insertBefore(this.createLineNumSpan(i + 1), lineElement.firstChild);
   }

What is the loop doing?

[hallvard]

The lineElements are span elements representing lines, wrapping restructured elements from highlightjs. The loop inserts line number elements first in each line element's child list. An example of what someone would like to do. This could be a build-in feature, e.g. an attribute on the code element that is picked up by the plugin.

[joshgoebel]

Looks interesting. Whenever you think you have something ready and generally useful feel free to add it to our Wiki: https://github.com/highlightjs/highlight.js/wiki/Useful-Plugins

For now we've decided it's best to use the wiki to maintain a list of plugins rather than host these repos here directly. We'll have to gradually bump it's prominence a bit as it isn't something we've taken advantage much in the past.

---

I almost wonder if you should perhaps consider the idea of being a "data plugin" though... limiting yourself to merely attaching the line data to either the element (or just the result object literal)... and then letting someone write a second plugin (using the official API) that could take advantage of the data you add in it's own after:highlightElement hook... just an idea though. It might make your plugin slightly simpler and decouple it from how exactly people wanted to use it.

The callback approach might be simpler for implementors though. Sadly we don't have a lot of great patterns built up around the use of plugins yet.

[hallvard]

Now I've implemented both the callback approach, which is very general, and some specific behaviour for adding line numbers using a format string and adding class names to elements. You're right we could just leave it to others to operate on the result, but after reading about how highlightjs never would support line numbers, I though it would be nice to at least support that. And I needed the class name feature myself, so ended up with those two.

Concerning "chaining" plugins: Is the order of calling plugins well-defined, so you can ensure the lineify plugin is run before any other that relies on the restructured html?

I'll add an entry to the wiki. I've improved the README slightly, one thing missing is how to include/import it. Currently, it isn't published anywhere, so you must either refer to it using a gitcontent url or save it to a local file, like I guess some do for highlightjs. I've no experience publishing, but I guess npm has a task for it.

[joshgoebel]

Is the order of calling plugins well-defined, so you can ensure the lineify plugin is run before any other that relies on the restructured html?

Installing plugins in the proper order is the duty of the implementer. Callbacks hooks are executed against plugins in the order that the plugins were added... if someone wanting to build on your plugin would need to install it before their own.

I've no experience publishing, but I guess npm has a task for it.

It's not super hard to publish an npm package once you have an account... there are docs/tutorials for it if you look.