Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add KaTeX rendering to Markdown. #20571

Merged
merged 36 commits into from
Sep 13, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
f57fb78
Add KaTeX rendering to Markdown.
zeripath Jul 22, 2021
3ac5fb5
Switch to always available rendering with ```math
zeripath Aug 2, 2022
1f81006
remove goldmark meta and handle yaml frontmatter ourselves
zeripath Aug 2, 2022
fa26c58
missedremoval
zeripath Aug 2, 2022
1d6a332
placate lint
zeripath Aug 2, 2022
3a8c4f0
Merge remote-tracking branch 'origin/main' into katex-rendering
zeripath Aug 27, 2022
11c5c8d
Merge branch 'main' into katex-rendering
zeripath Sep 1, 2022
eba0331
Merge remote-tracking branch 'origin/main' into katex-rendering
zeripath Sep 2, 2022
a775a9c
as per silverwind
zeripath Sep 2, 2022
ca3573d
Apply suggestions from code review
zeripath Sep 3, 2022
810c790
placate lint
zeripath Sep 3, 2022
606ff7e
Merge remote-tracking branch 'origin/main' into katex-rendering
zeripath Sep 3, 2022
3b361e2
Handle conflict from #20987
zeripath Sep 3, 2022
438f9fa
Set inline dollar math to true - to match github
zeripath Sep 4, 2022
47127fe
Merge remote-tracking branch 'origin/main' into katex-rendering
zeripath Sep 4, 2022
056ab70
fix inline placement of display math
zeripath Sep 4, 2022
de68952
Merge branch 'main' into katex-rendering
lunny Sep 4, 2022
7a2387a
Merge branch 'main' into katex-rendering
zeripath Sep 5, 2022
5289f35
Merge branch 'main' into katex-rendering
zeripath Sep 6, 2022
0cb8346
Merge branch 'main' into katex-rendering
zeripath Sep 7, 2022
b45a6fa
Merge branch 'main' into katex-rendering
zeripath Sep 7, 2022
39d7b47
Merge branch 'main' into katex-rendering
lunny Sep 8, 2022
a7d60af
Merge branch 'main' into katex-rendering
lunny Sep 11, 2022
841a22c
Merge branch 'main' into katex-rendering
silverwind Sep 11, 2022
563a798
Merge branch 'main' into katex-rendering
lunny Sep 11, 2022
f114106
remove inline dollar option
zeripath Sep 11, 2022
4b138db
fix spelling bug
zeripath Sep 11, 2022
92d3748
comments
zeripath Sep 11, 2022
e7851ca
as per delvh
zeripath Sep 11, 2022
07c13ce
Merge branch 'main' into katex-rendering
zeripath Sep 11, 2022
97dcc1f
Merge branch 'main' into katex-rendering
lunny Sep 12, 2022
8eae07a
Merge branch 'main' into katex-rendering
zeripath Sep 12, 2022
59e0b7f
docs update
silverwind Sep 12, 2022
b53d5cb
Merge branch 'main' into katex-rendering
zeripath Sep 12, 2022
caa4130
Merge branch 'main' into katex-rendering
zeripath Sep 12, 2022
3409860
Merge branch 'main' into katex-rendering
lunny Sep 13, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions custom/conf/app.example.ini
Original file line number Diff line number Diff line change
Expand Up @@ -1262,6 +1262,9 @@ ROUTER = console
;; List of file extensions that should be rendered/edited as Markdown
;; Separate the extensions with a comma. To render files without any extension as markdown, just put a comma
;FILE_EXTENSIONS = .md,.markdown,.mdown,.mkd
;;
;; Enables math inline and block detection
;ENABLE_MATH = true

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
Expand Down
1 change: 1 addition & 0 deletions docs/content/doc/advanced/config-cheat-sheet.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ The following configuration set `Content-Type: application/vnd.android.package-a
- `CUSTOM_URL_SCHEMES`: Use a comma separated list (ftp,git,svn) to indicate additional
URL hyperlinks to be rendered in Markdown. URLs beginning in http and https are
always displayed
- `ENABLE_MATH`: **true**: Enables detection of `\(...\)`, `\[...\]`, `$...$` and `$$...$$` blocks as math blocks.

## Server (`server`)

Expand Down
3 changes: 2 additions & 1 deletion docs/content/doc/advanced/external-renderers.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,12 +74,13 @@ RENDER_COMMAND = "timeout 30s pandoc +RTS -M512M -RTS -f rst"
IS_INPUT_FILE = false
```

If your external markup relies on additional classes and attributes on the generated HTML elements, you might need to enable custom sanitizer policies. Gitea uses the [`bluemonday`](https://godoc.org/github.com/microcosm-cc/bluemonday) package as our HTML sanitizier. The example below will support [KaTeX](https://katex.org/) output from [`pandoc`](https://pandoc.org/).
If your external markup relies on additional classes and attributes on the generated HTML elements, you might need to enable custom sanitizer policies. Gitea uses the [`bluemonday`](https://godoc.org/github.com/microcosm-cc/bluemonday) package as our HTML sanitizer. The example below could be used to support server-side [KaTeX](https://katex.org/) rendering output from [`pandoc`](https://pandoc.org/).

```ini
[markup.sanitizer.TeX]
; Pandoc renders TeX segments as <span>s with the "math" class, optionally
; with "inline" or "display" classes depending on context.
; - note this is different from the built-in math support in our markdown parser which uses <code>
ELEMENT = span
ALLOW_ATTR = class
REGEXP = ^\s*((math(\s+|$)|inline(\s+|$)|display(\s+|$)))+
Expand Down
2 changes: 2 additions & 0 deletions docs/content/doc/features/comparison.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ _Symbols used in table:_
| WebAuthn (2FA) | ✓ | ✘ | ✓ | ✓ | ✓ | ✓ | ? |
| Built-in CI/CD | ✘ | ✘ | ✓ | ✓ | ✓ | ✘ | ✘ |
| Subgroups: groups within groups | [✘](https://github.com/go-gitea/gitea/issues/1872) | ✘ | ✘ | ✓ | ✓ | ✘ | ✓ |
| Mermaid diagrams in Markdown | ✓ | ✘ | ✓ | ✓ | ✓ | ✘ | ✘ |
| Math syntax in Markdown | ✓ | ✘ | ✓ | ✓ | ✓ | ✘ | ✘ |

## Code management

Expand Down
3 changes: 2 additions & 1 deletion docs/content/page/index.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,8 @@ You can try it out using [the online demo](https://try.gitea.io/).
- Environment variables
- Command line options
- Multi-language support ([21 languages](https://github.com/go-gitea/gitea/tree/main/options/locale))
- [Mermaid](https://mermaidjs.github.io/) Diagram support
- [Mermaid](https://mermaidjs.github.io/) diagrams in Markdown
- Math syntax in Markdown
- Mail service
- Notifications
- Registration confirmation
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ require (
gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df
gopkg.in/ini.v1 v1.67.0
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
mvdan.cc/xurls/v2 v2.4.0
strk.kbt.io/projects/go/libravatar v0.0.0-20191008002943-06d1c002b251
xorm.io/builder v0.3.11
Expand Down Expand Up @@ -290,7 +291,6 @@ require (
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
gopkg.in/cheggaaa/pb.v1 v1.0.28 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
sigs.k8s.io/yaml v1.2.0 // indirect
)

Expand Down
84 changes: 84 additions & 0 deletions modules/markup/markdown/convertyaml.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package markdown

import (
"github.com/yuin/goldmark/ast"
east "github.com/yuin/goldmark/extension/ast"
"gopkg.in/yaml.v3"
)

func nodeToTable(meta *yaml.Node) ast.Node {
for {
if meta == nil {
return nil
}
switch meta.Kind {
case yaml.DocumentNode:
meta = meta.Content[0]
continue
default:
}
break
}
switch meta.Kind {
case yaml.MappingNode:
return mappingNodeToTable(meta)
case yaml.SequenceNode:
return sequenceNodeToTable(meta)
default:
return ast.NewString([]byte(meta.Value))
}
}

func mappingNodeToTable(meta *yaml.Node) ast.Node {
table := east.NewTable()
alignments := []east.Alignment{}
for i := 0; i < len(meta.Content); i += 2 {
alignments = append(alignments, east.AlignNone)
}

headerRow := east.NewTableRow(alignments)
valueRow := east.NewTableRow(alignments)
for i := 0; i < len(meta.Content); i += 2 {
cell := east.NewTableCell()

cell.AppendChild(cell, nodeToTable(meta.Content[i]))
headerRow.AppendChild(headerRow, cell)

if i+1 < len(meta.Content) {
cell = east.NewTableCell()
cell.AppendChild(cell, nodeToTable(meta.Content[i+1]))
valueRow.AppendChild(valueRow, cell)
}
}

table.AppendChild(table, east.NewTableHeader(headerRow))
table.AppendChild(table, valueRow)
return table
}

func sequenceNodeToTable(meta *yaml.Node) ast.Node {
table := east.NewTable()
alignments := []east.Alignment{east.AlignNone}
for _, item := range meta.Content {
row := east.NewTableRow(alignments)
cell := east.NewTableCell()
cell.AppendChild(cell, nodeToTable(item))
row.AppendChild(row, cell)
table.AppendChild(table, row)
}
return table
}

func nodeToDetails(meta *yaml.Node, icon string) ast.Node {
details := NewDetails()
summary := NewSummary()
summary.AppendChild(summary, NewIcon(icon))
details.AppendChild(details, summary)
details.AppendChild(details, nodeToTable(meta))

return details
}
15 changes: 3 additions & 12 deletions modules/markup/markdown/goldmark.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ import (
"code.gitea.io/gitea/modules/setting"
giteautil "code.gitea.io/gitea/modules/util"

meta "github.com/yuin/goldmark-meta"
"github.com/yuin/goldmark/ast"
east "github.com/yuin/goldmark/extension/ast"
"github.com/yuin/goldmark/parser"
Expand All @@ -32,20 +31,12 @@ type ASTTransformer struct{}

// Transform transforms the given AST tree.
func (g *ASTTransformer) Transform(node *ast.Document, reader text.Reader, pc parser.Context) {
metaData := meta.GetItems(pc)
firstChild := node.FirstChild()
createTOC := false
ctx := pc.Get(renderContextKey).(*markup.RenderContext)
rc := &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "",
}

if metaData != nil {
rc.ToRenderConfig(metaData)

metaNode := rc.toMetaNode(metaData)
rc := pc.Get(renderConfigKey).(*RenderConfig)
if rc.yamlNode != nil {
metaNode := rc.toMetaNode()
if metaNode != nil {
node.InsertBefore(node, firstChild, metaNode)
}
Expand Down
20 changes: 18 additions & 2 deletions modules/markup/markdown/markdown.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/markup"
"code.gitea.io/gitea/modules/markup/common"
"code.gitea.io/gitea/modules/markup/markdown/math"
"code.gitea.io/gitea/modules/setting"
giteautil "code.gitea.io/gitea/modules/util"

Expand All @@ -38,6 +39,7 @@ var (
isWikiKey = parser.NewContextKey()
renderMetasKey = parser.NewContextKey()
renderContextKey = parser.NewContextKey()
renderConfigKey = parser.NewContextKey()
)

type limitWriter struct {
Expand Down Expand Up @@ -98,7 +100,7 @@ func actualRender(ctx *markup.RenderContext, input io.Reader, output io.Writer)
languageStr := string(language)

preClasses := []string{"code-block"}
if languageStr == "mermaid" {
if languageStr == "mermaid" || languageStr == "math" {
preClasses = append(preClasses, "is-loading")
}

Expand All @@ -120,6 +122,9 @@ func actualRender(ctx *markup.RenderContext, input io.Reader, output io.Writer)
}
}),
),
math.NewExtension(
math.Enabled(setting.Markdown.EnableMath),
),
meta.Meta,
),
goldmark.WithParserOptions(
Expand Down Expand Up @@ -167,7 +172,18 @@ func actualRender(ctx *markup.RenderContext, input io.Reader, output io.Writer)
log.Error("Unable to ReadAll: %v", err)
return err
}
if err := converter.Convert(giteautil.NormalizeEOL(buf), lw, parser.WithContext(pc)); err != nil {
buf = giteautil.NormalizeEOL(buf)

rc := &RenderConfig{
Meta: "table",
Icon: "table",
Lang: "",
}
buf, _ = ExtractMetadataBytes(buf, rc)

pc.Set(renderConfigKey, rc)

if err := converter.Convert(buf, lw, parser.WithContext(pc)); err != nil {
log.Error("Unable to render: %v", err)
return err
}
Expand Down
42 changes: 42 additions & 0 deletions modules/markup/markdown/math/block_node.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.

package math

import "github.com/yuin/goldmark/ast"

// Block represents a display math block e.g. $$...$$ or \[...\]
type Block struct {
ast.BaseBlock
Dollars bool
Indent int
Closed bool
}

// KindBlock is the node kind for math blocks
var KindBlock = ast.NewNodeKind("MathBlock")

// NewBlock creates a new math Block
func NewBlock(dollars bool, indent int) *Block {
return &Block{
Dollars: dollars,
Indent: indent,
}
}

// Dump dumps the block to a string
func (n *Block) Dump(source []byte, level int) {
m := map[string]string{}
ast.DumpHelper(n, source, level, m, nil)
}

// Kind returns KindBlock for math Blocks
func (n *Block) Kind() ast.NodeKind {
return KindBlock
}

// IsRaw returns true as this block should not be processed further
func (n *Block) IsRaw() bool {
return true
}
Loading