feat: Refactor api-markdown-documenter to be written in terms of AS…

…Ts and functional transformations (#14201)

## BREAKING CHANGE
This is a complete overhaul of the library, greatly changing its API
surface. The general consumption patterns remain mostly the same, but
the largely operate on our new AST types.

## New Architecture

The old architecture was based on (and greatly re-used) the base
API-Documenter library. This PR moves mostly away from this. Rather than
operating on a flow that resembled something like:
```
ApiItem (w/ DocNode) -> DocNode -> stateful render
```

We now transform in input ApiItem tree (generated by API-Extractor) into
an [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree)-based
Documentation Domain. From there, we provide a more-functional rendering
library for rendering that domain as Markdown. So something more like:
```
ApiItem -> Documentation Domain -> functional render
```

### Benefits

- Functional renderer is much easier to understand, easy to unit test
(see the `RenderFoo` test modules), and made it easy to clean up a
number of rendering inconsistencies hidden by the API-Documenter-based
`MarkdownEmitter` renderer.
- Intermediary Documentation Domain is easy to understand, `AST`
compatible, and requires no auxiliary state to process (old DocNode
domain required a tsdoc context to be able to walk, and also had to
reach back into the ApiItem domain to resolve links, etc.).
- Uses existing [unist](https://github.com/syntax-tree/unist) library
for AST representation (with domain-specific type-safety).

## Notes for reviewers

The actual code diff is probably not all that useful. Some of the
configuration types will look roughly the same, but otherwise the
majority of the code has been re-written.

I would recommend referring to the README's architectural overview /
changes to the prescribed workflow.

Additionally, you can view the diffs in our rendering snapshot tests to
confirm that behavioral changes are relatively small, and in my opinion,
are all improvements.

---------

Co-authored-by: Justin Myhres <[email protected]>
Co-authored-by: Justin Myhres <[email protected]>

.vscode/settings.json

@@ -33,6 +33,7 @@
         "pseudorandomly",
         "Tinylicious",
         "tombstoned",
+        "TSDoc",
         "unaugmented"
     ],
 

tools/api-markdown-documenter/.eslintrc.js

@@ -4,16 +4,15 @@
  */
 
 module.exports = {
-	extends: [require.resolve("@fluidframework/eslint-config-fluid/strict")],
+	extends: [require.resolve("@fluidframework/eslint-config-fluid/strict"), "prettier"],
 	parserOptions: {
 		project: ["./tsconfig.json"],
 	},
 	rules: {
 		/**
 		 * This package utilizes internals of api-documenter that are not exported by the package root.
 		 *
-		 * TODO: file issue to expose node types, etc. in main package so we don't need to do this, and have better
-		 * guarantees about support.
+		 * TODO: remove once we have completely migrated off of this library.
 		 */
 		"import/no-internal-modules": [
 			"error",
@@ -24,23 +23,28 @@ module.exports = {
 
 		"unicorn/prefer-module": "off",
 		"unicorn/prefer-negative-index": "off",
+		"unicorn/no-array-push-push": "off",
 
 		// This package is exclusively used in a Node.js context
 		"import/no-nodejs-modules": "off",
+	},
+	overrides: [
+		{
+			// Overrides for test files
+			files: ["src/**/test/**"],
+			plugins: ["chai-expect", "chai-friendly"],
+			extends: ["plugin:chai-expect/recommended", "plugin:chai-friendly/recommended"],
+			rules: {
+				"import/no-extraneous-dependencies": [
+					"error",
+					{
+						devDependencies: true,
+					},
+				],
 
-		// TODO: remove once dependency on base config has been updated.
-		"@typescript-eslint/explicit-member-accessibility": [
-			"error",
-			{
-				accessibility: "explicit",
-				overrides: {
-					accessors: "explicit",
-					constructors: "explicit",
-					methods: "explicit",
-					properties: "explicit",
-					parameterProperties: "explicit",
-				},
+				// Handled by chai-friendly instead.
+				"@typescript-eslint/no-unused-expressions": "off",
 			},
-		],
-	},
+		},
+	],
 };

tools/api-markdown-documenter/.mocharc.js

@@ -5,8 +5,8 @@
 
 "use strict";
 
-const config = {
-	require: ["@fluidframework/mocha-test-setup"],
-};
+const getFluidTestMochaConfig = require("@fluidframework/mocha-test-setup/mocharc-common");
 
+const packageDir = __dirname;
+const config = getFluidTestMochaConfig(packageDir);
 module.exports = config;

tools/api-markdown-documenter/.npmignore

@@ -1,6 +1,6 @@
 nyc
 *.log
 **/*.tsbuildinfo
-src/test
-dist/test
+src/**/test
+dist/**/test
 **/_api-extractor-temp/**

tools/api-markdown-documenter/.prettierignore

@@ -1,11 +1,11 @@
-# Generated by npm / Lerna
+# Generated by npm
 package-lock.json
 
 # Build output
 dist/*
 
 # Test output
-src/test/snapshots/*
+src/**/test/snapshots/*
 
 # Dependencies
 node_modules/*
@@ -14,8 +14,8 @@ node_modules/*
 _api-extractor-temp/*
 api-report/*
 
-# Generated documentation
-generated-api-docs/*
-
 # Generated type-tests
 **/*.generated.ts
+
+# Test coverage reports
+nyc/*

tools/api-markdown-documenter/README.md

@@ -2,6 +2,10 @@
 
 Contains a programmatic API for generating [Markdown][] documentation from an API report generated by [API-Extractor][].
 
+> Note: this library is specifically configured to target [GitHub flavored Markdown][].
+>
+> Compatibility with other Markdown flavors is not guaranteed, though you can always write your own renderer implementation against our [Documentation Domain](#documentation-domain)!
+
 It is similar to [API-Documenter][] and is heavily based upon it and uses it under the hood, but is designed to be more extensible and can be used programmatically.
 
 **Note**: this library does not currently offer a Command Line Interface (CLI).
@@ -27,7 +31,7 @@ npm i @fluid-tools/api-markdown-documenter -D
 
 ## Usage
 
-### Quickstart
+### Quick Start
 
 This library is intended to be highly customizable.
 That said, it is also intended to be easy to use out of the box.
@@ -38,20 +42,20 @@ Are you already generating `.api.json` report files as a part of your build?
 If yes, create a file called `api-markdown-documenter.js` and paste the following code:
 
 ```javascript
-const { readModel, renderFiles } = require("@fluid-tools/api-markdown-documenter");
+const { loadModel, renderApiModelAsMarkdown } = require("@fluid-tools/api-markdown-documenter");
 
 const inputDir = "<PATH-TO-YOUR-DIRECTORY-CONTAINING-API-REPORTS>";
 const outputDir = "<YOUR-OUTPUT-DIRECTORY-PATH>";
 
 // Create the API Model from our API reports
-const apiModel = await readModel(inputDir);
+const apiModel = await loadModel(inputDir);
 
 const config = {
 	apiModel,
 	uriRoot: ".",
 };
 
-await renderFiles(config, outputDir);
+await renderApiModelAsMarkdown(config, outputDir);
 ```
 
 The above script can be invoked as an `npm` script by adding the following to your `package.json`'s `scripts` property:
@@ -67,17 +71,22 @@ For more advanced usage options, see the following sections.
 
 This package contains 2 primary, programmatic entry-points for generating documentation:
 
-#### renderDocuments
+#### transformApiModel
 
-The `renderDocuments` function accepts an [ApiModel][] representing the package(s) of a repository, and generates a sequence of "Document" objects representing the resulting documentation based on the other provided configuration options.
+The `transformApiModel` function accepts an [ApiModel][] representing the package(s) of a repository, and generates a sequence of "Document" [Abstract Syntax Tree][] objects representing the resulting documentation based on the other provided configuration options.
 These objects include information about the page item, its documentation contents, and the intended output file path the document file should be rendered to, based on provided policy options.
 
+-   These trees are backed by [unist][]'s AST model.
+
 The input `ApiModel` here will generally be the output of [API-Extractor][].
 
-#### renderFiles
+See [Documentation Domain](#documentation-domain) below for more details on the output format.
+
+#### renderApiModelAsMarkdown
+
+The `renderApiModelAsMarkdown` function operates like [transformApiModel](#createdocuments), but writes the resulting documents to disk as files based on the provided configuration options.
 
-The `renderFiles` function operates like [renderDocuments](#renderdocuments), but writes the resulting documents to disk as files based on the provided configuration options.
-This function also accepts a `MarkdownEmitter` object that does the conversion from `DocNode` trees to a Markdown stream that will ultimately be rendered to file.
+This function accepts overrides for all of its default `Markdown`-rendering behaviors, so feel free to customize as you see fit!
 
 ### Loading the API Model
 
@@ -87,10 +96,10 @@ To generate an API model from `.api.json` files generated by `API-Extractor`, se
 
 ### Emitting Markdown Content
 
-If you are using the [renderDocuments](#renderdocuments) option described above, one option for emitting Markdown string content is to use the `emitMarkdown` function.
+If you are using the [transformApiModel](#createdocuments) option described above, one option for emitting Markdown string content is to use the `emitMarkdown` function.
 It accepts a `MarkdownDocument` object as input, and outputs a string representing the final Markdown content.
 
-Note: you can also accomplish this by just using [renderFiles](#renderfiles) if you are using default configuration / emitter options.
+Note: you can also accomplish this by just using [renderApiModelAsMarkdown](#renderfiles) if you are using default configuration / emitter options.
 
 ### Configuration Options
 
@@ -107,25 +116,98 @@ The vast majority of these options have default values that have been crafted to
 -   `uriRoot`: This is the root URI under which the generated documentation will be published.
     It will be used when generating links for API content docs.
 
+## Architectural Overview
+
+As noted above, this library is intended to be consumed programmatically.
+While we may at some point add some command line interfaces for common paths, the primary goal of this library's architecture is to be flexible and extensible.
+To that end, we have broken its logic into a few isolable steps, each offering its own extensibility offerings.
+
+End to end, this library can be viewed as a psuedo-functional transformation pipeline mapping from an [APIModel][] generated by `API-Extractor` to one or more `Markdown`-formatted document files.
+But this is broken into the following internal sequences:
+
+```mermaid
+graph LR
+    A[ApiModel]
+    B[Documentation AST]
+    C[Markdown]
+
+    A -- transformApiModel --> B
+    B -- renderDocumentAsMarkdown --> C
+    A -.- renderApiModelAsMarkdown -.-> C
+```
+
+For more details on the interior `Documentation AST` ([Abstract Syntax Tree][]) domain, see [Documentation Domain](#documentation-domain) below.
+
+### API-Extractor
+
+The input to our system is the [ApiModel][] generated by [API-Extractor][].
+
+This library offers the `loadModel` function as an option for loading your model from the generated `.api.json` metadata files `API-Extractor` generates be default.
+
+To transform this input to our [Documentation Domain][], you may call the `transformApiModel` function.
+This function walks the input `ApiModel` tree, transforming each `ApiItem` under it according to its configured series of transformation policies.
+These policies are entirely configurable, though the system offers defaults out of the box.
+
+### Documentation Domain
+
+This library defines its own "Documentation Domain" using [Abstract Syntax Tree][] (AST) syntax backed by [unist][].
+This is used as an intermediate domain between the `API-Extractor`-based input, and the `Markdown` rendering output.
+This domain was crafted to support [TSDoc][]'s capabilities, and to represent something resembling lowest common denominator between `Markdown` and `HTML` syntax.
+
+As this domain is implemented as an `AST`, it is highly customizable.
+If you are interested in creating your own intermediate domain concepts, feel free to implement them.
+So long as you provide a corresponding [renderer policy](#markdown-renderer), the system will gladly accept them!
+
+### Markdown Renderer
+
+The final component of this library's transformation pipeline is its `Markdown` renderer.
+
+> Note: by default, this renderer is configured to generate [GitHub flavored Markdown][].
+>
+> Compatibility with other Markdown flavors is not guaranteed.
+
+As with the other logic in this library, the renderer is highly configurable.
+It will accept any `Documentation Domain` tree as input, and transform each node according to its configured render policies.
+
+If you would like to add rendering support for a custom `Documentation Domain` node type, simply provide a renderer policy associated with that node's `type` value.
+
+If you would like to change any or all of this library's default rendering policies, you may simply override the default policies for the desired `type`s.
+
 ## Upcoming Work
 
+-   Simplify input configurations
+    -   Separate configs for each pipeline segment
+    -   Clean up terminology ("policies" in particular is too vague - we should remove this)
+-   Add extensibility options for `DocNode` transformations
+    -   If a consumer has a custom tsdoc config associated with their API-Extractor setup, this will be needed.
+
+### Known Bugs
+
+-   Example titles are not respected. See TSDoc [@example spec](https://tsdoc.org/pages/tags/example/) - text on the same line as the tag should be treated as the example title.
+-   Types that extend or implement types with generic parameters result in signatures rendered with missing closing `>`s.
+
 ### Documentation Improvements
 
 -   Intro sandbox (api report)
+-   Extensibility examples (maybe use the AlertNode concept, once we move it out of the library and into the FluidFramework website build)
 
-### Styling improvements
+### Styling Improvements
 
--   Remove leading blank line in documents
--   Excessive blank lines in Signature sections
 -   Fix links to the same file (only need heading component, not file path)
     -   This will require plumbing down a context document item, so we can easily determine if the document to which the link is being generated is the same as the document being linked to.
+-   Config options for parsing TSDoc block comment contents as Markdown (and don't escape the contents)?
+-   Add support for Table Cell alignment
+
+### Performance Improvements
+
+-   Rather than repeatedly walking up a given `ApiItem`'s hierarchy when evaluating paths, links, etc., we could pass down transformation context object containing a running record of the item's hierarchy as we walk the tree.
 
 ## Longer-term work
 
--   Replace DocNode output / MarkdownEmitter with Markdown AST trees and simple interface for rendering those trees to a stream
 -   Support placing documents _within_ their own hierarchy (support for the "index" model used by systems like DocFX)
 -   Pre-canned policies (flat, index, adjacency)
--   Handle multiple package entrypoints
+-   Handle multiple package entry-points
+-   Add separate HTML transformation path, for consumers that want to go straight to HTML
 
 <!-- AUTO-GENERATED-CONTENT:START (README_CONTRIBUTION_GUIDELINES_SECTION:includeHeading=TRUE) -->
 
@@ -192,6 +274,10 @@ Use of Microsoft trademarks or logos in modified versions of this project must n
 <!-- Links -->
 
 [markdown]: https://en.wikipedia.org/wiki/Markdown
+[abstract syntax tree]: https://en.wikipedia.org/wiki/Abstract_syntax_tree
 [api-extractor]: https://api-extractor.com
 [api-documenter]: https://github.com/microsoft/rushstack/tree/main/apps/api-documenter
 [apimodel]: https://api.rushstack.io/pages/api-extractor-model.apimodel
+[github flavored markdown]: https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/about-writing-and-formatting-on-github
+[tsdoc]: https://tsdoc.org/
+[unist]: https://github.com/syntax-tree/unist

tools/api-markdown-documenter/api-extractor.json

@@ -1,6 +1,6 @@
 {
 	"$schema": "https://developer.microsoft.com/json-schemas/api-extractor/v7/api-extractor.schema.json",
-	"extends": "@fluidframework/build-common/api-extractor-common-strict.json",
+	"extends": "@fluidframework/build-common/api-extractor-common-report.json",
 	"apiReport": {
 		"enabled": true,
 		"reportFolder": "<projectFolder>/api-report/"