[Logs UI] Use Kibana index patterns in the source configuration #88607
Labels
Feature:Logs Settings
Feature:Logs UI
Logs UI feature
Team:obs-ux-logs
Observability Logs User Experience Team
Milestone
Summary
Kibana index patterns will be the basis for analyst-created runtime fields and related features. The Logs UI wants to make use of that and should therefore base its index configuration on Kibana index patterns.
Task breakdown
KIP support
Working with runtime fields
These tasks are quite speculative and highly depend on the outcome of the ongoing design and UX research.
Background
The indices read by the Logs UI are currently set as a plain Elasticsearch index ane pattern in the
logAlias
field of each source configuration saved object. While this is a flexible way to configure the set of indices, it makes integrating with the Kibana features that rely on Kibana index patterns very difficult. These include the aforementioned upcoming runtime fields, but also saved searches, visualizations and dashboards.The historic arguments for implementing the index configuration that way have partly become void and the Kibana team is making an effort (#78376) to remove remaining hurdles:
Complexity of the user flow
History: Relying on a plain index name pattern means that we can offer a working out-of the box experience to the user. This was in contrast to the jarring "no index pattern found" message that greeted a user in discover when setting up a new Kibana instance or space. It required the user to leave the app context and create a Kibana index pattern in the management section.
Outlook: With the upcoming introduction of an index pattern creation flyout (#84725) the user can create new index patterns without leaving the app context.
Integrity of the configuration
History: Kibana index patterns used to store their field list in their saved object, which had to be refreshed explicitly by a user in the management section. That meant that the field list might be out of date or inappropriate for a user's permissions, which in turn could lead to unexpected breakage in the Logs UI. Furthermore, a deletion of the Kibana index pattern in the management section could leave the Logs UI source configuration in an inconsistent state because it refers to a non-existant Kibana index pattern.
Outlook: The Kibana index patterns don't store the full field list anymore. Only overrides are stored in a saved object, which removes the need for manual refreshes and reduces the dependence on the configuring user's permissions. To prevent breaking the integrity of reference graphs, saved objects can by now also declare dependencies on other saved objects.
Proposed solution
Instead of entering an index name pattern into an input text field, the user would be able to select one of the existing Kibana index patterns. If no suitable index pattern was found, the user could create it via the Kibana index pattern creation flyout without leaving the app.
Impact on the user flow
Instead of being dropped right into the log stream, the user would be presented with a prompt to set up the source configuration. Although this onboarding step could be seen as an obstacle, it has the advantage of making the possibility to change the underlying indices visible to the user.
Impact on the settings UI
Where the settings UI currently offers a text input field to enter an index name pattern, a list of available index patterns is shown. The listed elements can be checked for suitability and only selectable if they fulfill criteria like "has a @timestamp date field" and "has a message text field". Using the flyout created in (#84725) we could also offer the ability for the user to immediately create a new index pattern if desired.
Impact on alerting
The log threshold alert definitions currently store a reference to the source configuration, which the executors load to look up the indices and fields to use. This could be adapted to look up the index name pattern and timestamp field in the Kibana index pattern with a few localized changes.
Impact on the anomaly detection integration
The anomaly detection jobs currently capture the index name patterns and fields at job creation time. After jobs have been created, the UI warns if the source configuration and the job configurations diverge at a later time. This could be adapted to look up the index name pattern and timestamp field in the Kibana index pattern with a few localized changes.
Possible index pattern validation criteria
@timestamp
date
fieldmessage
text
fieldImpact on the source configuration saved object
Instead of storing a string in the
logAlias
field the saved object would store a propertylogIndices
that contains an extensible descriptor:The current
logAlias
field could be migrated to aLogIndexNameReference
and continue to function as before even though the UI pushes the user towards using Kibana index patterns.Backwards compatibility
The migration strategy described above combined with the choice of the legacy index name option in the UI could make this a backwards-compatible enhancement.
Outlook
Related links
The text was updated successfully, but these errors were encountered: