Adds docs for kibana_system privileges (#161697)

Introduces a new `Security` Key Concepts page to the developer
documentation, with two sections:
1. API Authorization, adapted from
#160351 (comment).
2. `kibana_system` privilege guidelines, adapted from our team's
internal documentation. There are two notable changes from the previous
version:

1) Removal of internal discussions, as this now exists within a public
repository.
2) Recreated the existing mermaid diagram into a public Whimsical
diagram. The new docs system supports the latter, but not the former.


## Testing
To build this locally, run `./scripts/dev_docs` from a local checkout of
this PR. A server will eventually start on `http://localhost:3000` where
you can preview the changes.

## Screenshots


<img width="1632" alt="CleanShot 2023-07-12 at 08 32 23@2x"
src="https://github.com/elastic/kibana/assets/3493255/1b7c5053-a1bf-434f-bdfd-3eeeee692e18">


![CleanShot 2023-07-12 at 08 33
21@2x](https://github.com/elastic/kibana/assets/3493255/b15083af-f2b4-4daf-93a7-d6f43c73f3f2)

dev_docs/key_concepts/security.mdx

@@ -0,0 +1,100 @@
+---
+id: kibDevDocsSecurityIntro
+slug: /kibana-dev-docs/key-concepts/security-intro
+title: Security
+description: Maintaining Kibana's security posture
+date: 2023-07-11
+tags: ['kibana', 'dev', 'contributor', 'security']
+---
+
+Security is everyone's responsibility. This is inclusive of design, product, and engineering. The purpose of this guide is to give a high-level overview of security constructs and expectations.
+
+This guide covers the following topics:
+
+* [API authorization](#api-authorization)
+* [The `kibana_system` user](#the-kibana_system-user)
+
+## API authorization
+Kibana API routes do not have any authorization checks applied by default. This means that your APIs are accessible to anyone with valid credentials, regardless of their permissions. This includes users with no roles assigned.
+This on its own is insufficient, and care must be taken to ensure that only authorized users can invoke your endpoints.
+
+### Adding API authorization
+Kibana leverages <DocLink id="kibDevDocsSavedObjectsIntro" text="Saved Objects" /> for a majority of its persistence. The Saved Objects Service performs its own authorization checks, so if your API route is primarily a CRUD interface to Saved Objects, then your authorization needs are already met.
+This is also true for derivatives of the Saved Objects Service, such as the Alerting and Cases services.
+
+If your endpoint is not a CRUD interface to Saved Objects, then your route should include `access` tags to ensure that only authorized users can invoke your endpoint. This is **especially** important if your route does any of the following:
+1. Performs non-insignificant processing, causing load on the Kibana server.
+2. Calls Elasticsearch using the internal `kibana_system` user.
+3. Calls a third-party service.
+4. Returns any non-public information to the caller, such as system configuration or state.
+
+More information on adding `access` tags to your routes can be found temporarily in the [legacy documentation](https://www.elastic.co/guide/en/kibana/current/development-security.html#development-plugin-feature-registration)
+
+### Why not add `access` tags to all routes by default?
+Each authorization check that we perform involves a round-trip to Elasticsearch, so they are not as cheap as we'd like. Therefore, we want to keep the number of authorization checks we perform within a single route to a minimum.
+Adding an `access` tag to routes that leverage the Saved Objects Service would be redundant in most cases, since the Saved Objects Service will be performing authorization anyway.
+
+
+## The `kibana_system` user
+
+The Kibana server authenticates to Elasticsearch using the `elastic/kibana` [service account](https://www.elastic.co/guide/en/elasticsearch/reference/current/service-accounts.html#service-accounts-explanation). This service account has privileges that are equivilent to the `kibana_system` reserved role, whose descriptor is managed in the Elasticsearch repository ([source link](https://github.com/elastic/elasticsearch/blob/430cde6909eae12e1a90ac2bff29b71cbf4af18b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java#L58)).
+The vast majority of features will not require changes to the `kibana_system` user privileges. Changes to these privileges must be carefully considered, as we do not want the `kibana_system` account to have access to user data.
+
+### Guiding principals
+
+Consider these guidelines when reviewing changes to the `kibana_system` role descriptor.
+
+#### 1. Kibana should only access system and hidden indices
+
+- System indices are managed entirely by the stack, and should never be accessed by end users.
+- Hidden indices are _typically_ managed by the stack, but _may_ be accessed by end users.
+- Data indices are those which an end user could conceivably create on their own. As a general rule, users can create indices of any pattern so long as it does not begin with a \``.`\` (dot). Users can also create hidden indices, so it is important that documentation exists for any stack-managed hidden indices to reduce the chance of conflict with user-managed indices.
+
+Therefore, Kibana should not have access to non-system indices which do not begin with a \``.`\` (dot).
+
+Kibana should also not have the ability to modify system/hidden indices for which it is not the owner.
+
+##### Examples
+| Index Type | Allowed Permissions | Examples |
+|-------|--------|-----
+| User-defined data index | none | `my-data`, `kibana-metrics` |
+| System index not owned by Kibana | `read` | `.security` |
+| System indices owned by Kibana | `all` | `.kibana*`, '.fleet*' |
+
+#### Decision tree
+<DocWhimsical id="kibana-system-privilege-decision-tree-VTTGaTtjs9SnpbRNSg2Ptp" title="Decision tree" />
+
+##### Exceptions for telemetry
+Exceptions to this rule have been made in the past to aid in telemetry collection. This is not something we want to support long-term, but there aren't viable alternatives at the moment without a significant redesign. 
+
+##### Exceptions for Fleet package lifecycle management
+Fleet maintains the lifecycle of certain packages. These packages need to be upgraded during stack upgrades, and therefore have to happen in an automated fashion. The `kibana_system` user has elevated privileges to a set of **data incides** to facilitate this.
+
+If the set of indices managed by Fleet changes, we should ensure that they update [the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) to make note of index naming collisions.
+
+
+#### 2. Kibana should not have the ability to manage security constructs.
+
+This includes:
+- Users
+- Roles
+- Role Mappings
+
+### Rationale
+
+These guidelines exist for two primary reasons.
+
+#### Reduce impact of account compromise
+Compromised `kibana_system` credentials can severely impact an installation. We want to make sure that this doesn't become catastrophic. More privileges == more potential damage. We shouldn't add privileges unnecessarily. We should remove privileges as soon as they aren't needed anymore.
+
+Credentials can be compromised in a number of ways:
+1. Insecure storage (e.g. `kibana.yml`, a post-it note, etc.).
+2. Kibana server host compromise.
+3. Kibana server runtime compromise (e.g. RCE).
+
+#### Reduce impact of developer error
+Kibana allows engineers to call ES using different sets of credentials:
+1. `kibana_system` credentials. 
+2. End-user credentials.
+
+An authorization bypass could occur if an engineer accidentally uses the `kibana_system` credentials when they intended to use end-user credentials. See [Broken Access Control](https://owasp.org/Top10/A01_2021-Broken_Access_Control/).

nav-kibana-dev.docnav.json

@@ -78,6 +78,9 @@
         {
           "id": "kibBuildingBlocks"
         },
+        {
+          "id": "kibDevDocsSecurityIntro"
+        },
         {
           "id": "kibDevDocsSavedObjectsIntro",
           "label": "Saved objects"
@@ -482,4 +485,4 @@
       ]
     }
   ]
-}
+}