Move public SO APIs to SOM

[TinaHeiligers]

Moving the public saved objects (import/export/resolve_import_error) APIs from Core to the Saved Objects management plugin (SOM) could improve code organization and API management.

If the APIs allow us to leverage more platform features. E.g. we could use the file service to store the export, taskManager to schedule export as background tasks, and so on.

If we used the file service, we could remove the resolve_import_error API: this API exists only because we need to re-send the import file for error resolution. Storing the imported file would improve UX (especially for non-UI usage) because consumers only need to use import to resolve conflicts.

Rationale

1. Isolation of Concerns:

   • Core: should primarily handle fundamental services that plugins and solutions rely on.
   • Plugins: Implement features and add-on functionality to Kibana.

   This ensures a clear separation of responsibilities and better modularity.

2. API Path Structure:

   • Current path prefix: /api/saved_objects/
     • These APIs currently reside under a generic path, which can be misleading. SOM is more distinct and specific.
   • Proposed path prefix: /api/kibana/management/saved_objects/
     A dedicated path under the SOM plugin is more descriptive and aligns with the purpose of these APIs, making it easier for users and developers to discover and understand the APIs.

3. Deprecation and Transition:

   • Deprecation Strategy: Maintain the current API prefix (/api/saved_objects/) as deprecated to provide a transition period for users and integrations to migrate to the new path.
   • New Stable Path: Encourage usage of the new API prefix (/api/kibana/management/saved_objects/) for all future development, ensuring a smooth transition.

Implementation Steps

1. Move APIs to SO Management Plugin:

   • Move the API code for import, export, and resolve_import_error APIs from Core to the SO management plugin.
   • Keep the importer/exporter code in Core. Update import paths throughout Kibana.

2. Update API Paths:

   • Implement the new API endpoints under the /api/kibana/management/saved_objects/ path within the SO management plugin.
   • Maintain the current API paths as deprecated, ensuring they continue to function but with clear deprecation warnings.

3. Documentation and Communication:

   • Update all relevant documentation to reflect the new API paths.
   • Communicate the changes and deprecation timeline to users and developers through appropriate channels (e.g., release notes, migration guides, kibana contributors newsletter).

4. Backward Compatibility:

   • Implement backward compatibility for the current API paths to ensure existing integrations do not break immediately.
   • Monitor usage and ensure there isn't any use before removing them after an appropriate deprecation period.

Benefits

1. Modularity:

   • Improves code organization by isolating SO management-related APIs within a dedicated plugin.
   • Makes API maintenance and potential future enhancements easier.

2. Clarity and Usability:

   • Provides a more intuitive and descriptive API path structure, enhancing the developer experience.
   • Helps users quickly identify the purpose and scope of the APIs.

3. Future-proofing:

   • Establishes a clear path for future SOM-related features and APIs in a dedicated plugin context.

By moving the public saved objects APIs to the SO management plugin and updating their API paths, we align the system architecture with best practices for modularity and clarity, ensuring a more maintainable and user-friendly API structure.

Related code references

• SOM plugin: https://github.com/elastic/kibana/tree/main/src/plugins/saved_objects_management
• SO HTTP routes currently in core: https://github.com/elastic/kibana/tree/main/packages/core/saved-objects/core-saved-objects-server-internal/src/routes (only need to move import.ts, export.ts, and resolve_import_errors.ts).

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[jloleysens]

Moving the public saved objects (import/export/resolve_import_error)

Just checking: my understanding of the proposal is that our desired state for/by 9.0 is:

Deprecate the folllowing:

public, deprecated: /api/saved_objects/_import
public, deprecated: /api/saved_objects/_export
public, deprecated: /api/saved_objects/_resolve_import_errors

Introduce the following:

public: /api/kibana/management/saved_objects/_import
public: /api/kibana/management/saved_objects/_export

@pgayvallet mentioned once we move these to SO Management plugin a path is available to get rid of _resolve_import_errors entirely. Might be worth taking stock of any implications that might have for the new /api/kibana/management/saved_objects/_import

[TinaHeiligers]

@jloleysens I was thinking more along the lines of introducing new SOM APIs for each of them:

Step 1:

public: /api/kibana/management/saved_objects/_import
public: /api/kibana/management/saved_objects/_export
public: /api/kibana/management/saved_objects/_resolve_import_errors

Step 2:
Validate we can remove _resolve_import_errors and deprecate the API

Step 3:
Wait until deprecation period is over

Step 4:
Remove the api.

_resolve_import_errors is public and we can't remove it without some sort of deprecation notice first.

I'm not saying we need to wait 18 months. We also don't need to wait long, only as long as it takes to make sure it's not consumed at all.

[TinaHeiligers]

spin-off from https://github.com/elastic/dev/issues/2200 , cleaning up core

[TinaHeiligers]

Not urgent since we're not removing the SO HTTP APIs in v9.

[afharo]

@TinaHeiligers, just to clarify. This issue is "just" about moving the HTTP API definitions to the SOM plugin. No need for additional optimizations, correct?

[TinaHeiligers]

@afharo This issue is for moving the public export and import HTTP routes to SOM. Enhancements and optimizations aren't directly in scope.

[afharo]

Nice! Thanks! I'll add links to the relevant code then to make it more good first issue :)

[afharo]

Removing the label good first issue and moved to Clarifying because we noticed 2 side-effects that require further discussion:

1. Reporting usage of the APIs via coreUsageDataClient is no-longer possible. Do we want to use usageCounters instead?
2. The config savedObjects.maxImportPayloadBytes and savedObjects.maxImportExportSize must be moved to the SOM plugin (deprecations => BCC?) or exposed by the core service for the plugin to read.

[TinaHeiligers]

Related is to start working on documentation for migrating off of the existing, deprecated APIs.
ideally, we'd have that automated or at the very least, some tooling to help end users.
I'll be fleshing a concept that's stewing in more detail over the next months.