Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Manager] Split OpenAPI spec into multiple files #77378

Closed
wants to merge 6 commits into from

Conversation

jfsiii
Copy link
Contributor

@jfsiii jfsiii commented Sep 14, 2020

Summary

WIP. First pass at #77290

  • spec_entry.json is the version which links to the various files on disk.

  • spec_bundled.json is the resolved output of that entry & other files in one file. It's currently generated with

    npx swagger-cli bundle spec_entry.json -o spec_bundled.json

  • Paths: this defines each endpoint. A path can have one operation per http method.

    Went with "directory-per-path" approach which is largely "Each operation in a separate file" from paths/README.md. Only index.json at the moment but they can be split out to something like get.json, put.json or some-operation-id.json, have a local examples, etc.

    tree ./paths
    paths/
    ├── README.md
    ├── agent_policies
    │   ├── delete
    │   │   └── index.json
    │   ├── index.json
    │   └── {agent_policy_id}
    │       ├── copy
    │       │   └── index.json
    │       └── index.json
    ├── epm
    │   ├── categories
    │   │   └── index.json
    │   └── packages
    │       ├── index.json
    │       └── {pkgkey}
    │           └── index.json
    ├── fleet
    │   ├── agent-status
    │   │   └── index.json
    │   ├── agents
    │   │   ├── enroll
    │   │   │   └── index.json
    │   │   ├── index.json
    │   │   └── {agent_id}
    │   │       ├── acks
    │   │       │   └── index.json
    │   │       ├── checkin
    │   │       │   └── index.json
    │   │       ├── events
    │   │       │   └── index.json
    │   │       ├── index.json
    │   │       └── unenroll
    │   │           └── index.json
    │   ├── enrollment_api_keys
    │   │   ├── index.json
    │   │   └── {key_id}
    │   │       └── index.json
    │   ├── install
    │   │   └── {os_type}
    │   │       └── index.json
    │   └── setup
    │       └── index.json
    ├── package_policies
    │   ├── index.json
    │   └── {package_policy_id}
    │       └── index.json
    └── setup
        └── index.json
    
  • Components: Reusable components like schemas,
    responses
    parameters, etc

    tree ./components
    components/
    ├── README.md
    ├── callbacks
    ├── examples
    ├── headers
    ├── links
    ├── parameters
    │   ├── kuery_param.json
    │   ├── page_index_param.json
    │   ├── page_size_param.json
    │   └── xsrf_header.json
    ├── request_bodies
    ├── responses
    ├── schemas
    │   ├── access_api_key.json
    │   ├── agent.json
    │   ├── agent_event.json
    │   ├── agent_metadata.json
    │   ├── agent_policy.json
    │   ├── agent_status.json
    │   ├── agent_type.json
    │   ├── enrollment_api_key.json
    │   ├── new_agent_event.json
    │   ├── new_agent_policy.json
    │   ├── new_package_policy.json
    │   ├── package_info.json
    │   ├── package_policy.json
    │   └── search_result.json
    └── security_schemes
        ├── access_api_key.json
        ├── basic_auth.json
        └── enrollment_api_key.json
    

Checklist

Delete any items that are not applicable to this PR.

@jfsiii jfsiii self-assigned this Sep 14, 2020
@jfsiii jfsiii added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 14, 2020
@jfsiii
Copy link
Contributor Author

jfsiii commented Sep 14, 2020

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

distributable file count

id value diff baseline
default 45589 +44 45545

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jfsiii pushed a commit that referenced this pull request Oct 13, 2020
## Summary

replaces #77378 
fixes #77290

 * Replace the 4000+ line `spec_oas3.json` file with one ~75 line overview file `entrypoint.yaml` which imports the various pieces from other directories.
 * Switched from JSON to YAML for "source" files because they're less noisy & more human-friendly. `package-registry` & `package-spec` both define specs in YAML as well. We can always convert them to JS for any library that needs it.



## Dev docs
### New structure

```
openapi/
├── README.md
├── bundled.json
├── bundled.yaml
├── components/
├── entrypoint.yaml
└── paths/
```

There are `README.md` files in `openapi`, `openapi/components`, & `openapi/paths` with information about the purpose, conventions, decisions, etc for that directory

* `entrypoint.yaml` is the overview file which links to the various files on disk.
* `bundled.{yaml,json}` is the resolved output of that entry & other files in a single file. 
    <details><summary>how these were generated (requires node 12+)</summary>

    ```
    nvm use 12;
    npx @redocly/openapi-cli bundle entrypoint.yaml -o bundled.json
    npx @redocly/openapi-cli bundle entrypoint.yaml -o bundled.yaml
    ```
    </details>

    <details><summary>How to generate with node 10+</summary>

    ```
    npx swagger-cli bundle -o bundled.json -t json entrypoint.yaml
    npx swagger-cli bundle -o bundled.yaml -t yaml entrypoint.yaml
    ```
    </details>

 * [Paths](paths/README.md): Started with a flat `paths` directory with each path in a separate file. We can move on to a nested file-per-operation like #77378 or any other approach later

    <details><summary><code>tree ./paths</code></summary>

    ```
    paths/
    ├── README.md
    ├── agent_policies.yaml
    ├── [email protected]
    ├── agent_policies@{agent_policy_id}.yaml
    ├── agent_policies@{agent_policy_id}@copy.yaml
    ├── agent_status.yaml
    ├── agents.yaml
    ├── agents@bulk_upgrade.yaml
    ├── [email protected]
    ├── [email protected]
    ├── agents@{agent_id}.yaml
    ├── agents@{agent_id}@acks.yaml
    ├── agents@{agent_id}@checkin.yaml
    ├── agents@{agent_id}@events.yaml
    ├── agents@{agent_id}@unenroll.yaml
    ├── agents@{agent_id}@upgrade.yaml
    ├── enrollment_api_keys.yaml
    ├── enrollment_api_keys@{key_id}.yaml
    ├── [email protected]
    ├── [email protected]
    ├── epm@packages@{pkgkey}.yaml
    ├── install@{os_type}.yaml
    ├── package_policies.yaml
    ├── package_policies@{package_policy_id}.yaml
    └── setup.yaml
    ```
    </details>

  * [Components](components/README.md): Reusable components like [`schemas`](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#schemaObject), [`responses`](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#responseObject) [`parameters`](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#parameterObject), etc
    <details><summary><code>tree ./components</code></summary>

    ```
    components/
    ├── README.md
    ├── callbacks
    ├── examples
    ├── headers
    │   └── kbn_xsrf.yaml
    ├── links
    ├── parameters
    │   ├── kuery.yaml
    │   ├── page_index.yaml
    │   └── page_size.yaml
    ├── request_bodies
    ├── responses
    ├── schemas
    │   ├── access_api_key.yaml
    │   ├── agent.yaml
    │   ├── agent_event.yaml
    │   ├── agent_metadata.yaml
    │   ├── agent_policy.yaml
    │   ├── agent_status.yaml
    │   ├── agent_type.yaml
    │   ├── bulk_upgrade_agents.yaml
    │   ├── enrollment_api_key.yaml
    │   ├── new_agent_event.yaml
    │   ├── new_agent_policy.yaml
    │   ├── new_package_policy.yaml
    │   ├── package_info.yaml
    │   ├── package_policy.yaml
    │   ├── search_result.yaml
    │   └── upgrade_agent.yaml
    └── security_schemes
    ```    
</details>
jfsiii pushed a commit that referenced this pull request Oct 13, 2020
## Summary

replaces #77378 
fixes #77290

 * Replace the 4000+ line `spec_oas3.json` file with one ~75 line overview file `entrypoint.yaml` which imports the various pieces from other directories.
 * Switched from JSON to YAML for "source" files because they're less noisy & more human-friendly. `package-registry` & `package-spec` both define specs in YAML as well. We can always convert them to JS for any library that needs it.



## Dev docs
### New structure

```
openapi/
├── README.md
├── bundled.json
├── bundled.yaml
├── components/
├── entrypoint.yaml
└── paths/
```

There are `README.md` files in `openapi`, `openapi/components`, & `openapi/paths` with information about the purpose, conventions, decisions, etc for that directory

* `entrypoint.yaml` is the overview file which links to the various files on disk.
* `bundled.{yaml,json}` is the resolved output of that entry & other files in a single file. 
    <details><summary>how these were generated (requires node 12+)</summary>

    ```
    nvm use 12;
    npx @redocly/openapi-cli bundle entrypoint.yaml -o bundled.json
    npx @redocly/openapi-cli bundle entrypoint.yaml -o bundled.yaml
    ```
    </details>

    <details><summary>How to generate with node 10+</summary>

    ```
    npx swagger-cli bundle -o bundled.json -t json entrypoint.yaml
    npx swagger-cli bundle -o bundled.yaml -t yaml entrypoint.yaml
    ```
    </details>

 * [Paths](paths/README.md): Started with a flat `paths` directory with each path in a separate file. We can move on to a nested file-per-operation like #77378 or any other approach later

    <details><summary><code>tree ./paths</code></summary>

    ```
    paths/
    ├── README.md
    ├── agent_policies.yaml
    ├── [email protected]
    ├── agent_policies@{agent_policy_id}.yaml
    ├── agent_policies@{agent_policy_id}@copy.yaml
    ├── agent_status.yaml
    ├── agents.yaml
    ├── agents@bulk_upgrade.yaml
    ├── [email protected]
    ├── [email protected]
    ├── agents@{agent_id}.yaml
    ├── agents@{agent_id}@acks.yaml
    ├── agents@{agent_id}@checkin.yaml
    ├── agents@{agent_id}@events.yaml
    ├── agents@{agent_id}@unenroll.yaml
    ├── agents@{agent_id}@upgrade.yaml
    ├── enrollment_api_keys.yaml
    ├── enrollment_api_keys@{key_id}.yaml
    ├── [email protected]
    ├── [email protected]
    ├── epm@packages@{pkgkey}.yaml
    ├── install@{os_type}.yaml
    ├── package_policies.yaml
    ├── package_policies@{package_policy_id}.yaml
    └── setup.yaml
    ```
    </details>

  * [Components](components/README.md): Reusable components like [`schemas`](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#schemaObject), [`responses`](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#responseObject) [`parameters`](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#parameterObject), etc
    <details><summary><code>tree ./components</code></summary>

    ```
    components/
    ├── README.md
    ├── callbacks
    ├── examples
    ├── headers
    │   └── kbn_xsrf.yaml
    ├── links
    ├── parameters
    │   ├── kuery.yaml
    │   ├── page_index.yaml
    │   └── page_size.yaml
    ├── request_bodies
    ├── responses
    ├── schemas
    │   ├── access_api_key.yaml
    │   ├── agent.yaml
    │   ├── agent_event.yaml
    │   ├── agent_metadata.yaml
    │   ├── agent_policy.yaml
    │   ├── agent_status.yaml
    │   ├── agent_type.yaml
    │   ├── bulk_upgrade_agents.yaml
    │   ├── enrollment_api_key.yaml
    │   ├── new_agent_event.yaml
    │   ├── new_agent_policy.yaml
    │   ├── new_package_policy.yaml
    │   ├── package_info.yaml
    │   ├── package_policy.yaml
    │   ├── search_result.yaml
    │   └── upgrade_agent.yaml
    └── security_schemes
    ```    
</details>
@jfsiii
Copy link
Contributor Author

jfsiii commented Oct 13, 2020

was replaced by #80107

@jfsiii jfsiii closed this Oct 13, 2020
@jfsiii
Copy link
Contributor Author

jfsiii commented Oct 13, 2020

was replaced by #80107

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants