KQL/Kuery support for TSVB, timelion and Vega visualizations

[stacey-gammon]

Currently none of the above visualizations listen for KQL/Kuery queries, if that feature is enabled.

The reason for this is the usage of the broken dashboard_context for generating the DSL for a query. This plainly ignores if the query is actually written in KQL/Kquery.

This class should be removed completely and rather the correct way of implementing this should be used:

The request handler of a visualization will get passed in the filters and query in the params object (2nd argument to the request handler). Every visualization should use that query and filters instead of trying to access any global state (like dashboard_context does). Accessing global state in any visualization will break other features that we are planning like Custom time ranges per dashboard panel (#17776). It's an anti pattern that makes the whole architecture unstable and hard to maintain.

A visualization needs to look into filters/queries passed to it's request handler. To compile the query, there are APIs for Lucene and Kuery. We should imho look into creating one central "Query API" (cc @elastic/kibana-discovery), that accepts any query object and that itself will check what query type that query is and use the appropriate compiler. That way we don't duplicate that work into each individual visualization.

That compiled DSL needs then to be merged into the query done by the appropriate visualization. Since KQL/Kuery needs to be compiled against an index pattern, all visualizations need to figure out the appropriate index pattern, for their given index pattern string (maybe we could move that into the Query compile API?).

[timroes]

https://github.com/elastic/kibana/blob/master/src/core_plugins/kibana/public/dashboard/dashboard_context.js#L17:L21

[timroes]

@elastic/kibana-discovery I am not exactly aware how to work with KQL yet. We would need to open a PR that has an else if in there for KQL and converts it to ES query. Do you think you have capacity to open that, or could you give us the APIs needed for that?

[Bargs]

Dashboard should not treat the query like a filter anymore. The only reason that was necessary in the past was because queries did not get merged when the SearchSource hierarchy was flattened. I changed that back in 6.0, queries now get merged, and if my memory serves I also updated dashboard so it wouldn't put the query in the filters anymore. I'm not sure how this dashboard_context.js gets used, but it looks like it may have re-introduced that behavior. We also really need to stop passing around ES query DSL.

Also, as long as SearchSource is being used, no special handling of KQL is necessary.

[stacey-gammon]

I assigned myself, will look into it when I can but no promises on a soon-ish time.

[stacey-gammon]

This came up in a meeting today with @ppisljar, @timroes , @Bargs and @spalger.

Turns out there is no simple fix for this, and once we pass filters, queries, etc down from dashboard to embeddables directly, it will negate the need for this entire file. The problem with a quick fix is that even though there is an available conversion function available, something like fromKuery, it assumes a single index pattern, but those visualization types can use multiple index patterns.

So for now, we are living with this limitation since they are all experimental features, until we can clean up search source issues and pass down filters/queries, etc directly (issue pending for that goal).

[stacey-gammon]

Got rid of the bug label since technically we are considering this an unsupported feature.

[timroes]

Blocked by #20457

[nyurik]

From Vega's perspective, it needs an API to convert current dashboard's context to the MUST and MUST_NOT clauses for the ES query. This way Vega users can continue to write custom ES queries, and simply add %dashboard_context-must_clause% in the middle of their own query, which will be expanded with a list of dashboard-generated clauses. Vega authors already have to provide index pattern string.

[syberyjskimisiek]

Hi, kind of new here, I had a go at upgrading made by @PhaedrusTheGreek - transform_vis plugin. I removed dashboard context and replaced it with query/filters section, it installed sucessfully to my surprise however whenever I try to use second argument of myRequestHandler I somehow get "Cannot read property 'appState' of undefined" as a global error on a kibana when creating new vis. I cannot debug what's the actual argument passed to my custom Request Handler it comes either as undefined when the vis comes defined as an proper object. According to a documentation (from ES site for 6.6.0) it should be there along with uiState and timeRange? Or have I missed anything?

@PhaedrusTheGreek btw thanks mate this vizualisation of yours gives awesome flexibility in creating custom views :).

[syberyjskimisiek]

It seems that only one object is passed over (vis) not 2 as it was before, however documentations remains unchanged. I managed to launch transform_vis with success :). There are still a little tweaks to be done but I think it works now.

[Bargs]

This was fixed in #28010

[Classic92]

#17722 (comment)

I have the same problem like syberyjskimisiek
Is it possible to explain me this problem. I can install the Plugin transform_vis but i have also a problem with the appState. I can't see also the JavaScript section. Where is my mistake

[syberyjskimisiek]

@Classic92, if I remember correctly only one object is returned through myRequestHandler. This was the main change. You need to change your code to accept 1 object instead of 2, and somehow resolve the appState being undefined issue.