Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HTTP/OAS] Merge OpenAPI specs by using kbn-openapi-bundler #189262

Merged
merged 16 commits into from
Aug 13, 2024
Merged

[HTTP/OAS] Merge OpenAPI specs by using kbn-openapi-bundler #189262

merged 16 commits into from
Aug 13, 2024

Conversation

maximpn
Copy link
Contributor

@maximpn maximpn commented Jul 26, 2024 •
edited
Loading

Addresses: #186356
Relates to: #184428

Summary

This PR adds a merging JS script based on the utility implemented in #186356. Resulted OpenAPI bundle as committed in oas_docs/output/kibana.serverless.bundled.yaml.

Details

#188110 implements and exposes merge utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below

const { merge } = require('@kbn/openapi-bundler');

(async () => {
  await merge({
   sourceGlobs: [/* a list of source globs goes here */],
   outputFilePath: 'path/to/the/output/file.yaml',
  });
})();

The JS script added in this PR includes source OpenAPI specs presented in oas_docs/makefile plus Security Solution OpenAPI specs based on #184428.

To run the script use the following command from Kibana root folder

node ./oas_docs/scripts/merge_serverless_oas.js

Known linting issues with Security Solution OpenAPI specs

Running Spectral OpenAPI linter on the result bundle shows a number of errors caused by no-$ref-siblings rule. This caused by the current code generator implementation which requires default property to be set next to $ref though it's not correct for OpenAPI 3.0.3 while it's allowed in 3.1. It seems that Bump.sh handles such cases properly though by properly showing a default value.

We need to analyze the problem and decide if/when we should fix it.

The rest of warnings look fixable and will be addressed in the next stage after setting up linter rules.

Next steps

Since @kbn/openapi-bundler package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means oas_docs/makefile should be superseded by JS script(s) using merge utility form @kbn/openapi-bundler package.

@kbn/openapi-bundler SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting spectral-cli for linting purposes.

@maximpn maximpn added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc docs Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:OAS Work or issues related to Core-provided mechanisms for generating OAS v8.16.0 labels Jul 26, 2024
@maximpn maximpn requested review from jloleysens, lcawl and xcrzx July 26, 2024 09:48
@maximpn maximpn marked this pull request as ready for review July 26, 2024 09:48
@maximpn maximpn requested a review from a team as a code owner July 26, 2024 09:48
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@maximpn maximpn added the release_note:skip Skip the PR/issue when compiling release notes label Jul 26, 2024
@maximpn maximpn self-assigned this Jul 26, 2024
@maximpn maximpn force-pushed the merge-oas-with-kbn-openapi-bundler branch from cc41f04 to d625f11 Compare July 26, 2024 10:06
@maximpn maximpn changed the title [HTTP/OAS] Merge OpenAPI specs by using @kbn/openapi-bundler [HTTP/OAS] Merge OpenAPI specs by using kbn-openapi-bundler Jul 29, 2024
@maximpn maximpn force-pushed the merge-oas-with-kbn-openapi-bundler branch from d625f11 to dcc3530 Compare July 31, 2024 06:34
@maximpn maximpn removed request for xcrzx and a team July 31, 2024 06:36
@maximpn maximpn force-pushed the merge-oas-with-kbn-openapi-bundler branch 3 times, most recently from 6d781c3 to 210596e Compare August 2, 2024 15:00
@maximpn maximpn requested a review from a team as a code owner August 2, 2024 15:00
@maximpn maximpn force-pushed the merge-oas-with-kbn-openapi-bundler branch 2 times, most recently from 590f08e to fe987df Compare August 7, 2024 07:11
@maximpn maximpn requested a review from a team as a code owner August 7, 2024 07:11
jloleysens
jloleysens reviewed Aug 7, 2024
View reviewed changes
Copy link
Contributor

@jloleysens jloleysens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for driving this @maximpn ! I have previewed the output on bump.sh and it's looking mostly good. Left a few comments I'd like to get your thoughts on.

oas_docs/makefile
build-api-docs: ## Generate Serverless and ESS Kibana OpenAPI bundles with kbn-openapi-bundler
@node scripts/merge_serverless_oas.js
@node scripts/merge_ess_oas.js

.PHONY: api-docs
api-docs: ## Generate kibana.serverless.yaml and kibana.yaml
@npx @redocly/cli join "kibana.info.serverless.yaml" "../x-pack/plugins/observability_solution/apm/docs/openapi/apm.yaml" "../x-pack/plugins/actions/docs/openapi/bundled_serverless.yaml" "../src/plugins/data_views/docs/openapi/bundled.yaml" "../x-pack/plugins/ml/common/openapi/ml_apis_serverless.yaml" "../packages/core/saved-objects/docs/openapi/bundled_serverless.yaml" "../x-pack/plugins/observability_solution/slo/docs/openapi/slo/bundled.yaml" "bundle.serverless.json" -o "output/kibana.serverless.yaml" --prefix-components-with-info-prop title
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we replace this make command with the new bundler command? Or do we want to keep both? Maybe more a q for @lcawl , but curious to hear your thoughts too @maximpn

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want to remove existing command since @lcawl might use it. Eventually I got it reorganized to use OpenAPI bundler in api-docs command and renamed existing api-docs to api-docs-redocly. It's up to @lcawl to remove it if it's unused.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's good to leave for now since there are some outstanding issues (e.g. some operationIds that are not unique) that need to be resolved before we start publishing the security APIs. We can switch to the new command when those are fixed.

oas_docs/scripts/merge_serverless_oas.js Outdated Show resolved Hide resolved
oas_docs/output/kibana.merged.yaml Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see several entries of Elastic-Api-Version with values defined that don't fit the public API contract of a date string. I'm not sure where these values are coming from, could we just fallback to 2023-10-31 if not otherwise defined for all endpoints?

Full set of instances + line nr here 
68:            application/json; Elastic-Api-Version=0.2:
87:          application/json; Elastic-Api-Version=0.2:
160:          application/json; Elastic-Api-Version=0.2:
198:          application/json; Elastic-Api-Version=0.2:
215:            application/json; Elastic-Api-Version=0.2:
247:          application/json; Elastic-Api-Version=0.2:
267:            application/json; Elastic-Api-Version=0.2:
303:            application/json; Elastic-Api-Version=0.2:
384:            application/json; Elastic-Api-Version=0.2:
413:          application/json; Elastic-Api-Version=0.2:
424:            application/json; Elastic-Api-Version=0.2:
444:          application/json; Elastic-Api-Version=0.2:
455:            application/json; Elastic-Api-Version=0.2:
483:          application/json; Elastic-Api-Version=0.2:
515:            application/json; Elastic-Api-Version=0.2:
579:            application/json; Elastic-Api-Version=0.2:
602:            application/json; Elastic-Api-Version=0.2:
647:          application/json; Elastic-Api-Version=1.0.0:
664:            application/json; Elastic-Api-Version=1.0.0:
696:          application/json; Elastic-Api-Version=1.0.0:
719:            application/json; Elastic-Api-Version=1.0.0:
790:            application/json; Elastic-Api-Version=1.0.0:
1068:            application/json; Elastic-Api-Version=0.1:
1095:            application/json; Elastic-Api-Version=0.1:
1109:          application/json; Elastic-Api-Version=0.1:
1119:            application/json; Elastic-Api-Version=0.1:
1125:            application/json; Elastic-Api-Version=0.1:
1145:            application/json; Elastic-Api-Version=0.1:
1159:            application/json; Elastic-Api-Version=0.1:
1168:            application/json; Elastic-Api-Version=0.1:
1182:          application/json; Elastic-Api-Version=0.1:
1192:            application/json; Elastic-Api-Version=0.1:
1198:            application/json; Elastic-Api-Version=0.1:
1216:          application/json; Elastic-Api-Version=0.1:
1232:            application/json; Elastic-Api-Version=0.1:
1241:            application/json; Elastic-Api-Version=0.1:
1256:          application/json; Elastic-Api-Version=0.1:
1278:            application/json; Elastic-Api-Version=0.1:
1298:          application/json; Elastic-Api-Version=0.1:
1320:            application/json; Elastic-Api-Version=0.1:
1333:            application/json; Elastic-Api-Version=0.1:
1351:            application/json; Elastic-Api-Version=0.1:
1366:            application/json; Elastic-Api-Version=0.1:
1382:            application/json; Elastic-Api-Version=0.1:
1396:          application/json; Elastic-Api-Version=0.1:
1420:            application/json; Elastic-Api-Version=0.1:
1433:            application/json; Elastic-Api-Version=0.1:
1446:            application/json; Elastic-Api-Version=0.1:
1459:          application/json; Elastic-Api-Version=0.1:
1483:            application/json; Elastic-Api-Version=0.1:
1492:            application/json; Elastic-Api-Version=0.1:
1510:          application/json; Elastic-Api-Version=0.1:
1520:            application/json; Elastic-Api-Version=0.1:
1556:          application/json; Elastic-Api-Version=0.1:
1567:            application/json; Elastic-Api-Version=0.1:
3255:            application/json; Elastic-Api-Version=0.1:
3289:            application/json; Elastic-Api-Version=0.1:
3295:            application/json; Elastic-Api-Version=0.1:
7281:            application/json; Elastic-Api-Version=1.0.1:
7290:            application/json; Elastic-Api-Version=1.0.1:
7820:          application/json; Elastic-Api-Version=0.1:
7829:            application/json; Elastic-Api-Version=0.1:
7835:            application/json; Elastic-Api-Version=0.1:
7869:          application/json; Elastic-Api-Version=0.1:
7878:            application/json; Elastic-Api-Version=0.1:
7887:            application/json; Elastic-Api-Version=0.1:
7902:          application/json; Elastic-Api-Version=0.1:
7911:            application/json; Elastic-Api-Version=0.1:
7917:            application/json; Elastic-Api-Version=0.1:
7940:          application/json; Elastic-Api-Version=0.1:
7949:            application/json; Elastic-Api-Version=0.1:
7958:            application/json; Elastic-Api-Version=0.1:
7974:          application/json; Elastic-Api-Version=0.1:
7983:            application/json; Elastic-Api-Version=0.1:
7992:            application/json; Elastic-Api-Version=0.1:
8024:          application/json; Elastic-Api-Version=0.1:
8058:            application/x-ndjson; Elastic-Api-Version=0.1:
8068:            application/json; Elastic-Api-Version=0.1:
8194:            application/json; Elastic-Api-Version=0.1:
8200:            application/json; Elastic-Api-Version=0.1:
8259:          multipart/form-data; Elastic-Api-Version=0.1:
8278:            application/json; Elastic-Api-Version=0.1:
8324:            application/json; Elastic-Api-Version=0.1:
8373:          multipart/form-data; Elastic-Api-Version=0.1:
8438:            application/json; Elastic-Api-Version=0.1:
8482:            application/json; Elastic-Api-Version=0.1:
8504:          application/json; Elastic-Api-Version=0.1:
8520:            application/json; Elastic-Api-Version=0.1:
8526:            application/json; Elastic-Api-Version=0.1:
8544:            application/json; Elastic-Api-Version=0.1:
8550:            application/json; Elastic-Api-Version=0.1:
8574:          application/json; Elastic-Api-Version=0.1:
8590:            application/json; Elastic-Api-Version=0.1:
8596:            application/json; Elastic-Api-Version=0.1:
8613:          application/json; Elastic-Api-Version=0.1:
8620:            application/json; Elastic-Api-Version=0.1:
8626:            application/json; Elastic-Api-Version=0.1:
8632:            application/json; Elastic-Api-Version=0.1:
8657:            application/json; Elastic-Api-Version=0.1:
8663:            application/json; Elastic-Api-Version=0.1:
10132:            application/json; Elastic-Api-Version=1.0:
10138:            application/json; Elastic-Api-Version=1.0:
10144:            application/json; Elastic-Api-Version=1.0:
10150:            application/json; Elastic-Api-Version=1.0:
10156:            application/json; Elastic-Api-Version=1.0:
10173:          application/json; Elastic-Api-Version=1.0:
10180:            application/json; Elastic-Api-Version=1.0:
10186:            application/json; Elastic-Api-Version=1.0:
10192:            application/json; Elastic-Api-Version=1.0:
10198:            application/json; Elastic-Api-Version=1.0:
10204:            application/json; Elastic-Api-Version=1.0:
10223:          application/json; Elastic-Api-Version=1.0:
10232:            application/json; Elastic-Api-Version=1.0:
10238:            application/json; Elastic-Api-Version=1.0:
10244:            application/json; Elastic-Api-Version=1.0:
10266:            application/json; Elastic-Api-Version=1.0:
10272:            application/json; Elastic-Api-Version=1.0:
10278:            application/json; Elastic-Api-Version=1.0:
10284:            application/json; Elastic-Api-Version=1.0:
10309:            application/json; Elastic-Api-Version=1.0:
10315:            application/json; Elastic-Api-Version=1.0:
10321:            application/json; Elastic-Api-Version=1.0:
10327:            application/json; Elastic-Api-Version=1.0:
10333:            application/json; Elastic-Api-Version=1.0:
10351:          application/json; Elastic-Api-Version=1.0:
10358:            application/json; Elastic-Api-Version=1.0:
10364:            application/json; Elastic-Api-Version=1.0:
10370:            application/json; Elastic-Api-Version=1.0:
10376:            application/json; Elastic-Api-Version=1.0:
10382:            application/json; Elastic-Api-Version=1.0:
10402:            application/json; Elastic-Api-Version=1.0:
10408:            application/json; Elastic-Api-Version=1.0:
10414:            application/json; Elastic-Api-Version=1.0:
10420:            application/json; Elastic-Api-Version=1.0:
10426:            application/json; Elastic-Api-Version=1.0:
10448:            application/json; Elastic-Api-Version=1.0:
10454:            application/json; Elastic-Api-Version=1.0:
10460:            application/json; Elastic-Api-Version=1.0:
10466:            application/json; Elastic-Api-Version=1.0:
10488:            application/json; Elastic-Api-Version=1.0:
10494:            application/json; Elastic-Api-Version=1.0:
10500:            application/json; Elastic-Api-Version=1.0:
10506:            application/json; Elastic-Api-Version=1.0:
12724:        application/json; Elastic-Api-Version=0.2:
12730:        application/json; Elastic-Api-Version=0.2:
12750:        application/json; Elastic-Api-Version=0.2:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a known issue and there are multiple ways to solve it.

The right way to solve it is say the bundler what version source OpenAPI specs implement. One of the ideas is to use info.x-implementation-version in the source OpenAPI specs as in this example PR.

The simplest approach for now is to fallback to 2023-10-31 when it's impossible to detect an implementation version. It's not future proof since it won't work when new versions pop up.

Security Solution OpenAPI specs use info.version as an implementation version (it works for us since we consider document version equal to implementation version and wanna bump document version when we have incompatible changes). The bundler takes info.version (there are no other known version fields in OpenAPI specs) and uses it for Elastic-Api-Version.

@jloleysens what solution do you prefer?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The simplest approach for now is to fallback to 2023-10-31 when it's impossible to detect an implementation version. It's not future proof since it won't work when new versions pop up.

IMO this seems OK. In future we can find a way to provide version for a set of specs like a file naming my_api.2024-20-1.yaml or path convention /2024-20-1/my_api.yaml. The default would just be to use the "first" version. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good for me if @lcawl don't have any objections. I'll make adjustments in this PR (should be few lines change).

In future we can find a way to provide version for a set of specs like a file naming my_api.2024-20-1.yaml or path convention /2024-20-1/my_api.yaml. The default would just be to use the "first" version. WDYT?

Any option (or multiple options) works well if we agree on that across Kibana.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is difficult to visualize until we start having more than one Elastic-Api-Version value. However it's implemented in the code and the bundler, we'd ideally show all of the header values and their associated request and response bodies in a single document. As long as this PR is moving in that direction, it is sufficient for docs purposes.

jbudz
jbudz approved these changes Aug 7, 2024
View reviewed changes
Copy link
Member

@jbudz jbudz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.buildkite

@maximpn maximpn force-pushed the merge-oas-with-kbn-openapi-bundler branch 2 times, most recently from 5e8af02 to 9385e55 Compare August 9, 2024 08:16
@maximpn maximpn force-pushed the merge-oas-with-kbn-openapi-bundler branch from fc7903e to 4163443 Compare August 13, 2024 08:19
@maximpn maximpn requested a review from a team as a code owner August 13, 2024 08:24
banderror
banderror approved these changes Aug 13, 2024
View reviewed changes
Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rule Management-owned changes and PR overall LGTM 👍

One weird thing is that GitHub:

  • requested a review from @elastic/security-solution which is probably caused by changes in:
    • x-pack/plugins/security_solution/docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml
  • but didn't request a review from @elastic/security-detection-rule-management because of changes in:
    • packages/kbn-openapi-bundler/src/bundler/merge_documents/enrich_with_version_mime_param.ts
    • x-pack/plugins/security_solution/common/api/detection_engine/rule_management/bulk_crud/bulk_delete_rules/bulk_delete_rules_route.schema.yaml

@maximpn Could you please check if there's anything wrong with the CODEOWNERS file?

Apart from that, thanks for your hard work on Security OpenAPI specs and all the automation tools you created for us and other teams in Security. The progress on that is amazing and the quality of the solution is top-notch, especially considering the tight timelines and the cope of work that we had. Thank you! 👏

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @maximpn

@maximpn maximpn merged commit 7a2e7be into elastic:main Aug 13, 2024
39 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Aug 13, 2024
@maximpn maximpn deleted the merge-oas-with-kbn-openapi-bundler branch August 13, 2024 10:45
@maximpn maximpn mentioned this pull request Aug 13, 2024
Closed
4 tasks
@maximpn
Copy link
Contributor Author

maximpn commented Aug 13, 2024

@banderror

I checked CODEOWNERS file and haven't found anything suspicious. packages/kbn-openapi-bundler is assigned to @elastic/security-detection-rule-management as expected. The only thing we need to change there is assign domain OpenAPI bundles to appropriate teams instead of @elastic/security-solution .

Most probably GitHub's algorithm didn't handle review properly since I made changes to packages/kbn-openapi-bundler/src/bundler/merge_documents/enrich_with_version_mime_param.ts and
x-pack/plugins/security_solution/common/api/detection_engine/rule_management/bulk_crud/bulk_delete_rules/bulk_delete_rules_route.schema.yaml in the last commit. I'll keep an eye on that.

bryce-b pushed a commit to bryce-b/kibana that referenced this pull request Aug 13, 2024
@maximpn @bryce-b
[HTTP/OAS] Merge OpenAPI specs by using kbn-openapi-bundler (elasti…
bd61125 
…c#189262)

**Addresses:** elastic#186356
**Relates to:** elastic#184428

## Summary

This PR adds a merging JS script based on the utility implemented in elastic#186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`.

## Details

elastic#188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below

```js
const { merge } = require('@kbn/openapi-bundler');

(async () => {
  await merge({
   sourceGlobs: [/* a list of source globs goes here */],
   outputFilePath: 'path/to/the/output/file.yaml',
  });
})();
```

The JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on elastic#184428.

**To run** the script use the following command from Kibana root folder

```bash
node ./oas_docs/scripts/merge_serverless_oas.js 
```

## Known linting issues with Security Solution OpenAPI specs

Running Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value.

We need to analyze the problem and decide if/when we should fix it.

The rest of warnings look fixable and will be addressed in the next stage after setting up linter rules.

## Next steps

Since `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package.

`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.
This was referenced Aug 13, 2024
Merged
Merged
Merged
@lcawl lcawl mentioned this pull request Aug 14, 2024
Merged
@lcawl
Copy link
Contributor

lcawl commented Aug 14, 2024

💚 All backports created successfully

Status Branch Result
✅ 8.15

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@kibanamachine kibanamachine mentioned this pull request Aug 15, 2024
Merged
lcawl pushed a commit to lcawl/kibana that referenced this pull request Aug 20, 2024
@maximpn @lcawl
[HTTP/OAS] Merge OpenAPI specs by using kbn-openapi-bundler (elasti…
eee9ef8 
…c#189262)

**Addresses:** elastic#186356
**Relates to:** elastic#184428

This PR adds a merging JS script based on the utility implemented in elastic#186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`.

elastic#188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below

```js
const { merge } = require('@kbn/openapi-bundler');

(async () => {
  await merge({
   sourceGlobs: [/* a list of source globs goes here */],
   outputFilePath: 'path/to/the/output/file.yaml',
  });
})();
```

The JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on elastic#184428.

**To run** the script use the following command from Kibana root folder

```bash
node ./oas_docs/scripts/merge_serverless_oas.js
```

Running Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value.

We need to analyze the problem and decide if/when we should fix it.

The rest of warnings look fixable and will be addressed in the next stage after setting up linter rules.

Since `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package.

`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.
lcawl added a commit that referenced this pull request Aug 20, 2024
@lcawl @maximpn
[8.15] [HTTP/OAS] Merge OpenAPI specs by using kbn-openapi-bundler (#…
07194fd 
…189262) (#190541)

# Backport

This will backport the bundler aspects of the following commits from
`main` to `8.15`:
- [[HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler`
(#189262)](#189262)
 - #189621
 - #189482
 - #189348
 - #189472
 - #188110
 - #188812

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Maxim
Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-08-13T10:45:35Z","message":"[HTTP/OAS]
Merge OpenAPI specs by using `kbn-openapi-bundler`
(#189262)\n\n**Addresses:**
https://github.com/elastic/kibana/issues/186356\r\n**Relates to:**
https://github.com/elastic/kibana/issues/184428\r\n\r\n##
Summary\r\n\r\nThis PR adds a merging JS script based on the utility
implemented in #186356. Resulted
OpenAPI bundle as committed in
`oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n##
Details\r\n\r\nhttps://github.com//pull/188110 implements
and exposes `merge` utility design to merge source OpenAPI specs without
processing. It's has only a programmatic API. To merge OpenAPI specs
it's required to add a JS script like below\r\n\r\n```js\r\nconst {
merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n
await merge({\r\n sourceGlobs: [/* a list of source globs goes here
*/],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n
});\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes
source OpenAPI specs presented in `oas_docs/makefile` plus Security
Solution OpenAPI specs based on
https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the
script use the following command from Kibana root
folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js
\r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI
specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows
a number of errors caused by `no-$ref-siblings` rule. This caused by the
current code generator implementation which requires `default` property
to be set next to `$ref` though it's not correct for OpenAPI `3.0.3`
while it's allowed in `3.1`. It seems that Bump.sh handles such cases
properly though by properly showing a default value.\r\n\r\nWe need to
analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest
of warnings look fixable and will be addressed in the next stage after
setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince
`@kbn/openapi-bundler` package is tailored specifically for Kibana it
should replace Redocly currently used to merge OpenAPI specs. It also
means `oas_docs/makefile` should be superseded by JS script(s) using
`merge` utility form `@kbn/openapi-bundler`
package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI
linters since it doesn't perform thorough linting. It's good if we
continue adopting `spectral-cli` for linting
purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","backport:skip","docs","Team:
SecuritySolution","Feature:OAS","v8.16.0"],"number":189262,"url":"https://github.com/elastic/kibana/pull/189262","mergeCommit":{"message":"[HTTP/OAS]
Merge OpenAPI specs by using `kbn-openapi-bundler`
(#189262)\n\n**Addresses:**
https://github.com/elastic/kibana/issues/186356\r\n**Relates to:**
https://github.com/elastic/kibana/issues/184428\r\n\r\n##
Summary\r\n\r\nThis PR adds a merging JS script based on the utility
implemented in #186356. Resulted
OpenAPI bundle as committed in
`oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n##
Details\r\n\r\nhttps://github.com//pull/188110 implements
and exposes `merge` utility design to merge source OpenAPI specs without
processing. It's has only a programmatic API. To merge OpenAPI specs
it's required to add a JS script like below\r\n\r\n```js\r\nconst {
merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n
await merge({\r\n sourceGlobs: [/* a list of source globs goes here
*/],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n
});\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes
source OpenAPI specs presented in `oas_docs/makefile` plus Security
Solution OpenAPI specs based on
https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the
script use the following command from Kibana root
folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js
\r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI
specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows
a number of errors caused by `no-$ref-siblings` rule. This caused by the
current code generator implementation which requires `default` property
to be set next to `$ref` though it's not correct for OpenAPI `3.0.3`
while it's allowed in `3.1`. It seems that Bump.sh handles such cases
properly though by properly showing a default value.\r\n\r\nWe need to
analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest
of warnings look fixable and will be addressed in the next stage after
setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince
`@kbn/openapi-bundler` package is tailored specifically for Kibana it
should replace Redocly currently used to merge OpenAPI specs. It also
means `oas_docs/makefile` should be superseded by JS script(s) using
`merge` utility form `@kbn/openapi-bundler`
package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI
linters since it doesn't perform thorough linting. It's good if we
continue adopting `spectral-cli` for linting
purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/189262","number":189262,"mergeCommit":{"message":"[HTTP/OAS]
Merge OpenAPI specs by using `kbn-openapi-bundler`
(#189262)\n\n**Addresses:**
https://github.com/elastic/kibana/issues/186356\r\n**Relates to:**
https://github.com/elastic/kibana/issues/184428\r\n\r\n##
Summary\r\n\r\nThis PR adds a merging JS script based on the utility
implemented in #186356. Resulted
OpenAPI bundle as committed in
`oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n##
Details\r\n\r\nhttps://github.com//pull/188110 implements
and exposes `merge` utility design to merge source OpenAPI specs without
processing. It's has only a programmatic API. To merge OpenAPI specs
it's required to add a JS script like below\r\n\r\n```js\r\nconst {
merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n
await merge({\r\n sourceGlobs: [/* a list of source globs goes here
*/],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n
});\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes
source OpenAPI specs presented in `oas_docs/makefile` plus Security
Solution OpenAPI specs based on
https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the
script use the following command from Kibana root
folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js
\r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI
specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows
a number of errors caused by `no-$ref-siblings` rule. This caused by the
current code generator implementation which requires `default` property
to be set next to `$ref` though it's not correct for OpenAPI `3.0.3`
while it's allowed in `3.1`. It seems that Bump.sh handles such cases
properly though by properly showing a default value.\r\n\r\nWe need to
analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest
of warnings look fixable and will be addressed in the next stage after
setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince
`@kbn/openapi-bundler` package is tailored specifically for Kibana it
should replace Redocly currently used to merge OpenAPI specs. It also
means `oas_docs/makefile` should be superseded by JS script(s) using
`merge` utility form `@kbn/openapi-bundler`
package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI
linters since it doesn't perform thorough linting. It's good if we
continue adopting `spectral-cli` for linting
purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b"}}]}]
BACKPORT-->

---------

Co-authored-by: Maxim Palenov <[email protected]>
@maximpn maximpn mentioned this pull request Sep 19, 2024
Merged
maximpn added a commit that referenced this pull request Sep 22, 2024
@maximpn
[HTTP/OAS] Auto bundle staging OpenAPI specs (#193402)
a378064 
**Relates to:** #189262

## Summary

This PR adds scripts to automatically bundle staging Kibana OpenAPI specs in CI. It's done in a similar way as in #189262.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 22, 2024
@maximpn
[HTTP/OAS] Auto bundle staging OpenAPI specs (elastic#193402)
dd4cf5a 
**Relates to:** elastic#189262

## Summary

This PR adds scripts to automatically bundle staging Kibana OpenAPI specs in CI. It's done in a similar way as in elastic#189262.

(cherry picked from commit a378064)
kibanamachine referenced this pull request Sep 22, 2024
@kibanamachine @maximpn
[8.x] [HTTP/OAS] Auto bundle staging OpenAPI specs (#193402) (#193654)
6a9663f 
# Backport

This will backport the following commits from `main` to `8.x`:
- [[HTTP/OAS] Auto bundle staging OpenAPI specs
(#193402)](#193402)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Maxim
Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-22T07:15:57Z","message":"[HTTP/OAS]
Auto bundle staging OpenAPI specs (#193402)\n\n**Relates to:**
https://github.com/elastic/kibana/pull/189262\r\n\r\n##
Summary\r\n\r\nThis PR adds scripts to automatically bundle staging
Kibana OpenAPI specs in CI. It's done in a similar way as in
https://github.com/elastic/kibana/pull/189262.","sha":"a3780640060685f699687f1b9b9e55bd798fef07","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","v9.0.0","docs","Team:
SecuritySolution","backport:prev-minor","Feature:OAS","v8.16.0"],"title":"[HTTP/OAS]
Auto bundle staging OpenAPI
specs","number":193402,"url":"https://github.com/elastic/kibana/pull/193402","mergeCommit":{"message":"[HTTP/OAS]
Auto bundle staging OpenAPI specs (#193402)\n\n**Relates to:**
https://github.com/elastic/kibana/pull/189262\r\n\r\n##
Summary\r\n\r\nThis PR adds scripts to automatically bundle staging
Kibana OpenAPI specs in CI. It's done in a similar way as in
https://github.com/elastic/kibana/pull/189262.","sha":"a3780640060685f699687f1b9b9e55bd798fef07"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193402","number":193402,"mergeCommit":{"message":"[HTTP/OAS]
Auto bundle staging OpenAPI specs (#193402)\n\n**Relates to:**
https://github.com/elastic/kibana/pull/189262\r\n\r\n##
Summary\r\n\r\nThis PR adds scripts to automatically bundle staging
Kibana OpenAPI specs in CI. It's done in a similar way as in
https://github.com/elastic/kibana/pull/189262.","sha":"a3780640060685f699687f1b9b9e55bd798fef07"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: Maxim Palenov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lcawl lcawl lcawl approved these changes

@jloleysens jloleysens jloleysens approved these changes

@banderror banderror banderror approved these changes

@jbudz jbudz jbudz approved these changes

Assignees

@maximpn maximpn

Labels
backport:skip This commit does not require backporting docs Feature:OAS Work or issues related to Core-provided mechanisms for generating OAS release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.15.1 v8.16.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@maximpn @elasticmachine @kibana-ci @lcawl @jbudz @banderror @jloleysens @kibanamachine