dev: Add a file parser. (#1872)

* dev: cspell grammar

* test: add coverage

* dev: find first matching rule

* dev: Standardize on a single Rule context.

* dev: Progress towards parsing

* Update documentParser.ts

* Update grammarNormalizer.test.ts

* Update bin.js

* Create app.ts

* dev: a bit of refactoring

* dev: A bit of refactoring

* dev: Parse Simple Grammar

* dev: Update package-lock

cspell.code-workspace

@@ -90,10 +90,6 @@
     },
     "settings": {
         "typescript.tsdk": "cspell-monorepo/node_modules/typescript/lib",
-        "cSpell.files": [
-            "**",
-            "**/.*"
-        ],
         "cSpell.customDictionaries": {
             "workspace": true
         }

packages/cspell-grammar/.vscode/launch.json

@@ -0,0 +1,39 @@
+{
+    // Use IntelliSense to learn about possible Node.js debug attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "type": "node",
+            "request": "launch",
+            "name": "cspell-gitignore: Jest current-file",
+            "program": "${workspaceFolder}/../../node_modules/.bin/jest",
+            "args": [
+                "--runInBand",
+                "${fileBasename}"
+            ],
+            "console": "integratedTerminal",
+            "internalConsoleOptions": "neverOpen",
+            "disableOptimisticBPs": true,
+            "windows": {
+                "program": "${workspaceFolder}/../../node_modules/jest/bin/jest",
+            }
+        },
+        {
+            "type": "node",
+            "request": "launch",
+            "name": "cspell-gitignore: Jest All",
+            "program": "${workspaceFolder}/../../node_modules/.bin/jest",
+            "args": [
+                "--runInBand"
+            ],
+            "console": "integratedTerminal",
+            "internalConsoleOptions": "neverOpen",
+            "disableOptimisticBPs": true,
+            "windows": {
+                "program": "${workspaceFolder}/../../node_modules/jest/bin/jest",
+            }
+        }
+    ]
+}

packages/cspell-grammar/.vscode/tasks.json

@@ -0,0 +1,22 @@
+{
+    // See https://go.microsoft.com/fwlink/?LinkId=733558
+    // for the documentation about the tasks.json format
+    "version": "2.0.0",
+    "tasks": [
+        {
+            "type": "npm",
+            "script": "build",
+            "group": {
+                "kind": "build",
+                "isDefault": true
+            },
+            "problemMatcher": "$tsc"
+        },
+        {
+            "label": "Clean Build",
+            "type": "npm",
+            "script": "clean-build",
+            "problemMatcher": []
+        }
+    ]
+}

packages/cspell-grammar/CHANGELOG.md

@@ -0,0 +1,4 @@
+# Change Log
+
+All notable changes to this project will be documented in this file.
+See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

packages/cspell-grammar/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Jason Dent
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

packages/cspell-grammar/README.md

@@ -0,0 +1,96 @@
+# `cspell-grammar`
+
+CSpell Grammar is used define the parts of a document that is spell checked.
+
+In addition to being the cause of performance issues by end users, the existing `ignoreRegExpList` and `includeRegExpList` is too limited.
+
+The grammar is use to add `scope` to sections of a document. The `scope` can then be used to apply spell checking rules.
+
+Example: Only check comments and strings
+
+```yaml
+rules:
+  '*': false
+  comment: true
+  string: true
+```
+
+It can be even more powerful like controlling the language settings based upon scope.
+
+```yaml
+rules:
+  comment:
+    language: en
+  string:
+    language: en,fr
+    dictionaries: ['marketing-terms'],
+    caseSensitive: true
+  string.javascript:
+    caseSensitive: false
+```
+
+Rules are applied in the order they match the scope of the text.
+
+When checking JavaScript files:
+
+- strings will use the locale `en,fr`
+- the `marketing-terms` dictionary will be enabled
+- `caseSensitive` will be `false`
+
+At its core, `cspell-grammar` uses a simplified form of the TextMate grammar.
+
+## Reasoning
+
+Why use a grammar parser? Couldn't a colorizer / highlighter or a language AST be used?
+At one level, needs of the spell checker are simpler and different from colorizers or language AST parsers.
+The goal of a spell checker is to spell check **_relevant_** text. The spell check does not need to care about
+the syntactical correctness of a document or things like punctuation.
+
+The goal of a grammar parser for the spell checker is to allow the user to decide:
+
+1. What text should be checked.
+1. Which dictionaries (or languages) should be used.
+1. Are accents and case important
+
+Note: CSpell is a pure JavaScript application, so including the Oniguruma is not an option.
+
+### Considerations
+
+- Parsing a document should be fast - meaning the grammar should be as simple as possible to meet
+  the needs of the spell checker and not focus on scope detail. This is where a colorizer grammar is
+  not a good fit to be used.
+- AST's are a bit of an overkill for a spell checker. They provide too much detail while not bringing much benefit
+  from the detail.
+
+## Transformation
+
+Consider the following bit of LaTeX:
+
+```latex
+k\"{o}nnen
+können
+```
+
+<!--- cspell:ignore können nnen   -->
+
+For the spell checker to work correctly, the `\"{o}` should be transformed into `ö` before it is checked against the German dictionary.
+
+This creates a few challenges.
+
+Possible options:
+
+1. Simple whole document substitution
+   - Challenges
+     - It is not context aware and might replace the wrong text.
+     - It changes the location of the words and messes up issue reporting (some sort of Map would be needed to get the correct line / character offset).
+
+- Advantages
+  - Easy to implement except for the context and mapping.
+
+1. Scope level substitution
+   Transformations occur at the scope level.
+   - Challenges
+     - offset mapping is still and issue (maybe)
+     - need a way to merge text with adjacent scopes after transformation
+   - Advantages
+     - it is context aware

packages/cspell-grammar/bin.js

@@ -0,0 +1,5 @@
+'use strict';
+
+const app = require('./dist/app.js');
+
+app.run(process.argv);

packages/cspell-grammar/jest.config.js

@@ -0,0 +1 @@
+module.exports = require('../../jest.config');