Extract data format and flow docs into their own architecture documen…

…t. (#25299)

docs/architecture/data-flow.md

@@ -0,0 +1,129 @@
+# Data Flow and Data Format
+
+## The format
+
+A block editor post is the proper block-aware representation of a post: a collection of semantically consistent descriptions of what each block is and what its essential data is. This representation only ever exists in memory. It is the [chase](<https://en.wikipedia.org/wiki/Chase_(printing)>) in the typesetter's workshop, ever-shifting as [sorts](<https://en.wikipedia.org/wiki/Sort_(typesetting)>) are attached and repositioned.
+
+A block editor post is not the artifact it produces, namely the `post_content`. The latter is the printed page, optimized for the reader but retaining its invisible markings for later editing.
+
+The input and output of the block editor is a tree of block objects with the current format:
+
+```js
+const value = [
+    block1,
+    block2,
+    block3
+];
+```
+
+### The block object
+
+Each block object has an id, a set of attributes and potentially a list of child blocks.
+
+```js
+const block = {
+    clientId,   // unique string identifier.
+    type,       // The block type (paragraph, image...)
+    attributes, // (key, value) set of attributes representing the direct properties/content of the current block.
+    innerBlocks // An array of child blocks or inner blocks.
+}
+```
+
+Note the attributes keys and types, the allowed inner blocks are defined by the block type. For example, the core quote block has a `cite` string attribute reprensing the cite content while a heading block has a numeric `level` attribute, representing the level of the heading (1 to 6). 
+
+During the lifecycle of the block in the editor, the block object can receive extra metadata:
+
+ - `isValid`: A boolean representing whether the block is valid or not;
+ - `originalContent`: The original HTML serialization of the block.
+
+**Examples**
+
+```js
+// A simple paragraph block.
+const paragraphBlock1 = {
+    clientId: "51828be1-5f0d-4a6b-8099-f4c6f897e0a3",
+    type: "core/paragraph",
+    attributes: {
+        content: "This is the <strong>content</strong> of the paragraph block",
+        dropCap: true
+    }
+}
+
+// A separator block.
+const separatorBlock = {
+    clientId: "51828be1-5f0d-4a6b-8099-f4c6f897e0a4",
+    type: "core/separator",
+    attributes: {}
+}
+
+// A columns block with a paragraph block on each column.
+const columnsBlock = {
+    clientId: "51828be1-5f0d-4a6b-8099-f4c6f897e0a7",
+    type: "core/columns",
+    attributes: {},
+    innerBlocks: [
+        {
+            clientId: "51828be1-5f0d-4a6b-8099-f4c6f897e0a5",
+            type: "core/column",
+            attributes: {},
+            innerBlocks: [ paragraphBlock1 ],
+        },
+        {
+            clientId: "51828be1-5f0d-4a6b-8099-f4c6f897e0a6",
+            type: "core/column",
+            attributes: {},
+            innerBlocks: [ paragraphBlock2 ],
+        }
+    ]
+}
+```
+
+## Serialization and Parsing
+
+![Diagram](https://docs.google.com/drawings/d/1iuownt5etcih7rMMvPvh0Mny8zUA1Z28saxjxaWmfJ0/pub?w=1234&h=453)
+
+This data model, however, is something that lives in memory while editing a post. It's not visible to the page viewer when rendered, just like a printed page has no trace of the structure of the letters that produced it in the press.
+
+Since the whole WordPress ecosystem has an expectation for receiving HTML when rendering or editing a post, the block editor transforms its data into something that can be saved in `post_content` through serialization. This assures that there's a single source of truth for the content, and that this source remains readable and compatible with all the tools that interact with WordPress content at the present. Were we to store the object tree separately, we would face the risk of `post_content` and the tree getting out of sync and the problem of data duplication in both places.
+
+Thus, the serialization process converts the block tree into HTML using HTML comments as explicit block delimiters—which can contain the attributes in non-HTML form. This is the act of printing invisible marks on the printed page that leave a trace of the original structured intention.
+
+This is one end of the process. The other is how to recreate the collection of blocks whenever a post is to be edited again. A formal grammar defines how the serialized representation of a block editor post should be loaded, just as some basic rules define how to turn the tree into an HTML-like string. The block editor's posts aren't designed to be edited by hand; they aren't designed to be edited as HTML documents because the block editor posts aren't HTML in essence.
+
+They just happen, incidentally, to be stored inside of `post_content` in a way in which they require no transformation in order to be viewable by any legacy system. It's true that loading the stored HTML into a browser without the corresponding machinery might degrade the experience, and if it included dynamic blocks of content, the dynamic elements may not load, server-generated content may not appear, and interactive content may remain static. However, it at least protects against not being able to view block editor posts on themes and installations that are blocks-unaware, and it provides the most accessible way to the content. In other words, the post remains mostly intact even if the saved HTML is rendered as is.
+
+### Delimiters and Parsing Expression Grammar
+
+We chose instead to try to find a way to keep the formality, explicitness, and unambiguity in the existing HTML syntax. Within the HTML there were a number of options.
+
+Of these options, a novel approach was suggested: by storing data in HTML comments, we would know that we wouldn't break the rest of the HTML in the document, that browsers should ignore it, and that we could simplify our approach to parsing the document.
+
+Unique to HTML comments is the fact that they cannot legitimately exist in ambiguous places, such as inside of HTML attributes like `<img alt='data-id="14"'>`. Comments are also quite permissive. Whereas HTML attributes are complicated to parse properly, comments are quite easily described by a leading `<!--` followed by anything except `--` until the first `-->`. This simplicity and permissiveness means that the parser can be implemented in several ways without needing to understand HTML properly, and we have the liberty to use more convenient syntax inside of the comment—we only need to escape double-hyphen sequences. We take advantage of this in how we store block attributes: as JSON literals inside the comment.
+
+After running this through the parser, we're left with a simple object we can manipulate idiomatically, and we don't have to worry about escaping or unescaping the data. It's handled for us through the serialization process. Because the comments are so different from other HTML tags and because we can perform a first-pass to extract the top-level blocks, we don't actually depend on having fully valid HTML!
+
+This has dramatic implications for how simple and performant we can make our parser. These explicit boundaries also protect damage in a single block from bleeding into other blocks or tarnishing the entire document. It also allows the system to identify unrecognized blocks before rendering them.
+
+_N.B.:_ The defining aspects of blocks are their semantics and the isolation mechanism they provide: in other words, their identity. On the other hand, where their data is stored is a more liberal aspect. Blocks support more than just static local data (via JSON literals inside the HTML comment or within the block's HTML), and more mechanisms (_e.g._, global blocks or otherwise resorting to storage in complementary `WP_Post` objects) are expected. See [attributes](/docs/designers-developers/developers/block-api/block-attributes.md) for details.
+
+### The Anatomy of a Serialized Block
+
+When blocks are saved to the content after the editing session, its attributes—depending on the nature of the block—are serialized to these explicit comment delimiters.
+
+```html
+<!-- wp:image -->
+<figure class="wp-block-image"><img src="source.jpg" alt="" /></figure>
+<!-- /wp:image -->
+```
+
+A purely dynamic block that is to be server-rendered before display could look like this:
+
+```html
+<!-- wp:latest-posts {"postsToShow":4,"displayPostDate":true} /-->
+```
+
+## The Data Lifecycle
+
+In summary, the block editor workflow parses the saved document to an in-memory tree of blocks, using token delimiters to help. During editing, all manipulations happen within the block tree. The process ends by serializing the blocks back to the `post_content`.
+
+The workflow process relies on a serialization/parser pair to persist posts. Hypothetically, the post data structure could be stored using a plugin or retrieved from a remote JSON file to be converted to the block tree.

docs/architecture/key-concepts.md

@@ -0,0 +1,62 @@
+# Key Concepts
+
+## Blocks
+
+Blocks are an abstract unit for organizing and composing content, strung together to create content for a webpage.
+
+Blocks are hierarchical in that a block can be a child of or parent to another block. For example, a two-column Columns block can be the parent block to multiple child blocks in each of its columns.
+
+If it helps, you can think of blocks as a more graceful shortcode, with rich formatting tools for users to compose content. To this point, there is a new Block Grammar. Distilled, the block grammar is an HTML comment, either a self-closing tag or with a beginning tag and ending tag. In the main tag, depending on the block type and user customizations, there can be a JSON object. This raw form of the block is referred to as serialized.
+
+```html
+<!-- wp:paragraph {"key": "value"} -->
+<p>Welcome to the world of blocks.</p>
+<!-- /wp:paragraph -->
+```
+
+Blocks can be static or dynamic. Static blocks contain rendered content and an object of Attributes used to re-render based on changes. Dynamic blocks require server-side data and rendering while the post content is being generated (rendering).
+
+Each block contains Attributes or configuration settings, which can be sourced from raw HTML in the content via meta or other customizable origins.
+
+The Paragraph is the default block. Instead of a new line upon typing `return` on a keyboard, try to think of it as an empty Paragraph block (type "/" to trigger an autocompleting Slash Inserter -- "/image" will pull up Images as well as Instagram embeds).
+
+Users insert new blocks by clicking the plus button for the Block Inserter, typing "/" for the Slash Inserter, or typing `return` for a blank Paragraph block.
+
+Blocks can be duplicated within content using the menu from the block's toolbar or via keyboard shortcut.
+
+Blocks can also be made repeatable, allowing them to be shared across posts and post types and/or used multiple times in the same post. If a reusable block is edited in one place, those changes are reflected everywhere that that block is used.
+
+Blocks can be limited or locked-in-place by Templates and custom code.
+
+#### More on Blocks
+
+- **Block API**
+- **Block Styles**
+- **Tutorial: Building A Custom Block**
+
+## Block Categories
+
+In the Block Inserter (the accordion-sorted, popup modal that shows a site's available blocks to users) each accordion title is a Block Category, which are either the defaults or customized by developers through Plugins or code.
+
+## Reusable blocks
+
+Reusable blocks is a block (or multiple blocks) that you can insert, modify, repeatable piece of content.
+
+The content and style of a reusable block is intended to be consistent wherever it is used. 
+
+Examples of reusable blocks include a block consisting of a heading whose content and a custom color that would be appear on multiple pages of the site and sidebar widgets that would appear on every page (widgets are planned to be available, but not yet possible, in Gutenberg). 
+
+Any edits to a reusable block will appear on every other use of that block, saving time from having to make the same edit on different posts. 
+
+Reusable blocks are stored as a hidden post type (wp_block) and are dynamic blocks that "ref" or reference the post_id and return the post_content for that block. 
+
+The same reusable block can be used across different post types (e.g. post and page). 
+
+If you need to create a structure (a block consisting of heading, paragraph, and list) that is very similar across multiple posts but the content is slightly different across those pages or posts, you can do the following to minimize the amount of duplicate work to do:
+
+1. create a 'skeleton' that will have shared characteristics (e.g. the same color background, font size)
+1. save this as a reusable block.
+1. Then, on other pages/posts:
+1. Within the block editor: insert the reusable block 
+1. Open the block's properties (three dots)
+and "convert to regular block"; the block is no longer 'reusable' and all edits to this block will only appear on this page/post.

docs/architecture/readme.md

@@ -1,7 +1,9 @@
 # Architecture
 
-This living document serves as a high-level documentation to explain the conceptual choices and tool choices in the Gutenberg repository.
+Let’s look at the big picture and the architectural and UX principles of the block editor and the Gutenberg repository.
 
+- [Key Concepts](/docs/architecture/key-concepts.md)
+- [Data Format And Data Flow](/docs/architecture/data-flow.md)
 - [Understand the repository folder structure](/docs/architecture/folder-structure.md).
 - [Modularity and WordPress Packages](/docs/architecture/modularity.md).
 - [Block Editor Performance](/docs/architecture/performance.md).

docs/designers-developers/developers/tutorials/metabox/meta-block-1-intro.md

@@ -4,7 +4,7 @@ Typically, blocks store their attribute values in the serialised block HTML. How
 
 In this short tutorial you will create one of these blocks, which will prompt a user for a single value, and save it as post meta.
 
-For background around the thinking of blocks as the interface, please see the [Principles section](/docs/contributors/principles.md) of the handbook.
+For background around the thinking of blocks as the interface, please see the [key concepts section](/docs/architecture/key-concepts.md) of the handbook.
 
 Before starting this tutorial, you will need a plugin to hold your code. Please take a look at the first two steps of [the JavaScript tutorial](/docs/designers-developers/developers/tutorials/javascript/readme.md) for information setting up a plugin.