Sample data for Kibana

[alexfrancoeur]

One of the fastest ways to begin your journey with Kibana is to utilize sample data, dashboards and visualizations in order to familiarize yourself with the workflows and capabilities that the UI offers. We should give the user the option to use generic sample data when loading Kibana for the first time or whenever they'd like. As I see it, there are two approaches we can take.

1. Pre-package one or multiple small datasources with a separate download version of Kibana that includes demo data
2. Host sample data externally and provide a load option within the UI to choose from multiple sets of sample datasources

Option 1 would be easiest to implement

• A logical choice would be made by the user when downloading and setting up Kibana for the first time
• In order to remove the data, a user would have to manually delete the index, visualizations and dashboards associated with the index

Option 2 allows a user to be more flexible but would require more work to implement

• Users will have a choice as to what datasources / use cases they'd like demo data for
• Users can add sample data in at any time, not just their initial experience with Kibana
• UI modifications would need to be made to allow a user to choose what sample datasource they'd like to utilize
• If there is a dedicated UI for sample data, we could have a quick removal button to delete the index, visualizations and dashboards associated with the datasource
• An outbound connection would be necessary in order to load sample data, would need to gracefully handle a users experience if they are unable to load sample datasets

Sample data is crucial to improving the getting started experience for Kibana. We could either agree on an approach and execute or treat these as phases. Once discussed in more detail, we'll update the issue accordingly

[epixa]

Permissions for ingestion are the hard part here. Without an access control model in Kibana, we need rely on passing through Kibana user credentials to all ES requests, and the default x-pack security role for Kibana users does not include the ability to manage indices since those are pretty sensitive operations. This effectively rules out sample data ingested via human intervention via the UI itself, which makes a pretty big impact on the value here.

[tbragin]

@epixa If we simply ship a default dataset in a predefined Elasticsearch index and a sample Kibana dashboard, does that resolve the need to worry about permissions for ingestion? I thought this is where we landed last time we talked with @uboness @jimgoodwin and others.

[epixa]

@tbragin What do you mean by "ship a default dataset in a predefined Elasticsearch index"?

[tbragin]

Basically, ship the data with the stack, as opposed to on-demand download and index it. I think this is what Option 1 suggests.

[epixa]

So custom builds of Elasticsearch that include a custom data directory with the relevant indices/documents?

[uboness]

Option 2 is the ideal as far as I’m concerned… and also note that it’s not just about sample data, it may come with a bunch of predefined dashboards, saved searches, visualizations, etc... We’ve been discussing that if we had such a feature, we could take all the EDEN demos and fold it in there - effectively granting access to everyone to the use-case repository that maintain. That said, we need to think this thoroughly, and better understand the work required to implement this (also think the security implication of having this feature). Assuming Option 2 takes a long time to implement (which is just an assumption at this point) I’m not against Option 1 at all as an intermediate solution that later we can migrate to Option 2. That said, I’m not sure about the multiple or extra distributions for this purpose. Personally I wouldn’t mind adding this to our current build and have a simple link in the landing page that says “Load sample data and dashboards” (we can keep it small enough to effectively make little impact on bundle size).

[Bargs]

I suspect this feature is most valuable for people who have downloaded Kibana for the first time and just want to quickly play around with it to see what it's all about. In this situation they either have no security or they're probably logged in as the admin user. Couldn't we just show them this page if there's no security or they have index creation privileges?

[alexfrancoeur]

That said, I’m not sure about the multiple or extra distributions for this purpose. Personally I wouldn’t mind adding this to our current build and have a simple link in the landing page that says “Load sample data and dashboards” (we can keep it small enough to effectively make little impact on bundle size).

@uboness this might be the best approach for a first phase. It's a nice blend between options 1 and 2. We prepackage one "data source" that is already indexed in ES. In Kibana, we provide a landing page and / or quick link that simply creates an index pattern, visualizations and dashboards.

I see one potential issue with this. Ideally that link or button would need to be conditional based on whether or not the pre-defined index is available. Users could have manually removed this index or are using an older version of ES that did not have this sample data pre-packaged. Otherwise, we introduce another error during the getting started experience.

I also agree with @Bargs comment on showing / hiding a sample data source landing page (Eden-esque options) based on permissions.

[epixa]

We should pursue a better getting started experience regardless of what other plugins people have installed rather than assuming what situation a person may be in. I'm sure we can come up with a good solution here that works regardless of what plugins are installed though.

I see one potential issue with this. Ideally that link or button would need to be conditional based on whether or not the pre-defined index is available. Users could have manually removed this index or are using an older version of ES that did not have this sample data pre-packaged. Otherwise, we introduce another error during the getting started experience.

This seems OK to me. If the data doesn't exist, we don't show the getting started guide. We can even have people close the index as a way of "turning off" the getting started guide. This addresses the security concerns as well since it only shows up when you have permission to access the sample index.

Older versions of Elasticsearch shouldn't be an issue because Kibana can only run against the same version or above.

The biggest concern I have is the notion of bundling data into Elasticsearch by default for the sake of Kibana. That's a level of reverse coupling that we haven't really done before.

[ycombinator]

The biggest concern I have is the notion of bundling data into Elasticsearch by default for the sake of Kibana. That's a level of reverse coupling that we haven't really done before.

@epixa Perhaps I'm missing something but why couldn't we bundle the data with Kibana, not Elasticsearch? I'm imagining a Kibana server-side API that would load this data from disk (on the Kibana server) and call the ES bulk API to load it into a .kibana-sampledata (or similar) index. The Kibana API would use callWithRequest so user credentials (if they exist) would be passed to the ES request.

the default x-pack security role for Kibana users does not include the ability to manage indices since those are pretty sensitive operations. This effectively rules out sample data ingested via human intervention via the UI itself, which makes a pretty big impact on the value here.

What about augmenting the kibana_user role to have create_index, read, and write index-level privileges on the .kibana-sampledata index?