Migrating vega_vis from plugin (elastic#15014)

Large PR to migrate Vega plugin into the core.
https://github.com/nyurik/kibana-vega-vis

The code underwent a large number of changes and
reviews discussed in the PR thread:
elastic#15014

docs/visualize.asciidoc

@@ -50,6 +50,7 @@ data sets.
 <<tagcloud-chart,Tag cloud>>:: Display words as a cloud in which the size of the word correspond to its importance
 <<markdown-widget,Markdown widget>>:: Display free-form information or
 instructions.
+<<vega-graph,Vega graph>>:: Support for user-defined graphs, external data sources, images, and user-defined interactivity.
 . Specify a search query to retrieve the data for your visualization:
 ** To enter new search criteria, select the index pattern for the indices that
 contain the data you want to visualize. This opens the visualization builder
@@ -159,3 +160,5 @@ include::visualize/tagcloud.asciidoc[]
 include::visualize/heatmap.asciidoc[]
 
 include::visualize/visualization-raw-data.asciidoc[]
+
+include::visualize/vega.asciidoc[]

docs/visualize/vega.asciidoc

@@ -0,0 +1,289 @@
+[[vega-graph]]
+== Vega Graphs
+
+__________________________________________________________________________________________________________________________________________________________________________________
+Build https://vega.github.io/vega/examples/[Vega] and
+https://vega.github.io/vega-lite/examples/[VegaLite] data visualizations
+into Kibana.
+__________________________________________________________________________________________________________________________________________________________________________________
+
+[[vega-introduction-video]]
+=== Watch a short introduction video
+
+https://www.youtube.com/watch?v=lQGCipY3th8[image:https://i.ytimg.com/vi_webp/lQGCipY3th8/maxresdefault.webp[Kibana Vega Visualization Video]]
+
+[[vega-quick-demo]]
+=== Quick Demo
+
+* In Kibana, choose Visualize, and add Vega visualization.
+* You should immediately see a default graph
+* Try changing `mark` from `line` to `point`, `area`, `bar`, `circle`,
+`square`, ... (see
+https://vega.github.io/vega-lite/docs/mark.html#mark-def[docs])
+* Try other https://vega.github.io/vega/examples/[Vega] or
+https://vega.github.io/vega-lite/examples/[VegaLite] visualizations. You
+may need to make URLs absolute, e.g. replace
+`"url": "data/world-110m.json"` with
+`"url": "https://vega.github.io/editor/data/world-110m.json"`. (see
+link:#Using%20Vega%20and%20VegaLite%20examples[notes below])
+* Using https://www.npmjs.com/package/makelogs[makelogs util], generate
+some logstash data and try link:public/examples/logstash[logstash
+examples].  *(Do not use makelogs on a production cluster!)*
+
+[[vega-vs-vegalite]]
+=== Vega vs VegaLite
+
+VegaLite is a simplified version of Vega, useful to quickly get started,
+but has a number of limitations. VegaLite is automatically converted
+into Vega before rendering. Compare
+link:public/examples/logstash/logstash-simple_line-vega.json[logstash-simple_line-vega]
+and
+link:public/examples/logstash/logstash-simple_line-vegalite.json[logstash-simple_line-vegalite]
+(both use the same ElasticSearch logstash data). You may use
+https://vega.github.io/editor/[this editor] to convert VegaLite into
+Vega.
+
+[[vega-querying-elasticsearch]]
+== Querying ElasticSearch
+
+By default, Vega's https://vega.github.io/vega/docs/data/[data] element
+can use embedded and external data with a `"url"` parameter. Kibana adds support for the direct ElasticSearch queries by overloading
+the `"url"` value.
+
+Here is an example of an ES query that counts the number of documents in all indexes. The query uses *@timestamp* field to filter the time range, and break it into histogram buckets.
+
+[source,yaml]
+----
+// An object instead of a string for the url value
+// is treated as a context-aware Elasticsearch query.
+url: {
+  // Filter the time picker (upper right corner) with this field
+  %timefield%: @timestamp
+  // Apply dashboard context filters when set
+  %context%: true
+
+  // Which indexes to search
+  index: _all
+  // The body element may contain "aggs" and "query" subfields
+  body: {
+    aggs: {
+      time_buckets: {
+        date_histogram: {
+          // Use date histogram aggregation on @timestamp field
+          field: @timestamp
+          // interval value will depend on the daterange picker
+          // Use an integer to set approximate bucket count
+          interval: { %autointerval%: true }
+          // Make sure we get an entire range, even if it has no data
+          extended_bounds: {
+            min: { %timefilter%: "min" }
+            max: { %timefilter%: "max" }
+          }
+          // Use this for linear (e.g. line, area) graphs
+          // Without it, empty buckets will not show up
+          min_doc_count: 0
+        }
+      }
+    }
+    // Speed up the response by only including aggregation results
+    size: 0
+  }
+}
+----
+
+The full ES result has this kind of structure:
+
+[source,yaml]
+----
+{
+  "aggregations": {
+    "time_buckets": {
+      "buckets": [{
+          "key_as_string": "2015-11-30T22:00:00.000Z",
+          "key": 1448920800000,
+          "doc_count": 28
+        }, {
+          "key_as_string": "2015-11-30T23:00:00.000Z",
+          "key": 1448924400000,
+          "doc_count": 330
+        }, ...
+----
+
+Note that `"key"` is a unix timestamp, and can be used without conversions by the
+Vega date expressions.
+
+For most graphs we only need the list of the bucket values, so we use `format: {property: "aggregations.time_buckets.buckets"}` expression to focus on just the data we need.
+
+Query may be specified with individual range and dashboard context as
+well. This query is equivalent to `"%context%": true, "%timefield%": "@timestamp"`,
+except that the timerange is shifted back by 10 minutes:
+
+[source,yaml]
+----
+{
+  body: {
+    query: {
+      bool: {
+        must: [
+          // This string will be replaced
+          // with the auto-generated "MUST" clause
+          "%dashboard_context-must_clause%"
+          {
+            range: {
+              // apply timefilter (upper right corner)
+              // to the @timestamp variable
+              @timestamp: {
+                // "%timefilter%" will be replaced with
+                // the current values of the time filter
+                // (from the upper right corner)
+                "%timefilter%": true
+                // Only work with %timefilter%
+                // Shift current timefilter by 10 units back
+                shift: 10
+                // week, day (default), hour, minute, second
+                unit: minute
+              }
+            }
+          }
+        ]
+        must_not: [
+          // This string will be replaced with
+          // the auto-generated "MUST-NOT" clause
+          "%dashboard_context-must_not_clause%"
+        ]
+      }
+    }
+  }
+}
+----
+
+The `"%timefilter%"` can also be used to specify a single min or max
+value. As shown above, the date_histogram's `extended_bounds` can be set
+with two values - min and max. Instead of hardcoding a value, you may
+use `"min": {"%timefilter%": "min"}`, which will be replaced with the
+beginning of the current time range. The `shift` and `unit` values are
+also supported. The `"interval"` can also be set dynamically, depending
+on the currently picked range: `"interval": {"%autointerval%": 10}` will
+try to get about 10-15 data points (buckets).
+
+[[vega-esmfiles]]
+=== Elastic Map Files
+
+It is possible to access Elastic Map Service's files via the same mechanism
+
+[source,yaml]
+----
+url: {
+  // "type" defaults to "elasticsearch" otherwise
+  type: emsfile
+  // Name of the file, exactly as in the Region map visualization
+  name: World Countries
+}
+// The result is a geojson file, get its features to use
+// this data source with the "shape" marks
+// https://vega.github.io/vega/docs/marks/shape/
+format: {property: "features"}
+----
+
+[[vega-debugging]]
+== Debugging
+
+[[vega-browser-debugging-console]]
+=== Browser Debugging console
+
+Use browser debugging tools (e.g. F12 or Ctrl+Shift+J in Chrome) to
+inspect the `VEGA_DEBUG` variable:
+* `view` - access to the Vega View object. See https://vega.github.io/vega/docs/api/debugging/[Vega Debugging Guide]
+ on how to inspect data and signals at runtime. For VegaLite, `VEGA_DEBUG.view.data('source_0')` gets the main data set.
+ For Vega, it uses the data name as defined in your Vega spec.
+* `vega_spec` - Vega JSON graph specification after some modifications by Kibana. In case
+of VegaLite, this is the output of the VegaLite compiler.
+* `vegalite_spec` - If this is a VegaLite graph, JSON specification of the graph before
+VegaLite compilation.
+
+[[vega-data]]
+=== Data
+
+If you are using ElasticSearch query, make sure your resulting data is
+what you expected. The easiest way to view it is by using "networking"
+tab in the browser debugging tools (e.g. F12). Modify the graph slightly
+so that it makes a search request, and view the response from the
+server. Another approach is to use
+https://www.elastic.co/guide/en/kibana/current/console-kibana.html[Kibana
+Dev Tools] tab - place the index name into the first line:
+`GET <INDEX_NAME>/_search`, and add your query as the following lines
+(just the value of the `"query"` field)
+
+If you need to share your graph with someone, you may want to copy the
+raw data response to https://gist.github.com/[gist.github.com], possibly
+with a `.json` extension, use the `[raw]` button, and use that url
+directly in your graph.
+
+To restrict Vega from using non-ES data sources, add `vega.enableExternalUrls: false`
+to your kibana.yml file.
+
+[[vega-notes]]
+== Notes
+
+[[vega-useful-links]]
+=== Useful Links
+
+* https://vega.github.io/editor/[Editor] - includes examples for Vega &
+VegaLite, but does not support any Kibana-specific features like
+ElasticSearch requests and interactive base maps.
+* VegaLite
+https://vega.github.io/vega-lite/tutorials/getting_started.html[Tutorials],
+https://vega.github.io/vega-lite/docs/[docs], and
+https://vega.github.io/vega-lite/examples/[examples]
+* Vega https://vega.github.io/vega/tutorials/[Tutorial],
+https://vega.github.io/vega/docs/[docs],
+https://vega.github.io/vega/examples/[examples]
+
+[[vega-using-vega-and-vegalite-examples]]
+==== Using Vega and VegaLite examples
+
+When using https://vega.github.io/vega/examples/[Vega] and
+https://vega.github.io/vega-lite/examples/[VegaLite] examples, you may
+need to modify the "data" section to use absolute URL. For example,
+replace `"url": "data/world-110m.json"` with
+`"url": "https://vega.github.io/editor/data/world-110m.json"`. Also,
+regular Vega examples use `"autosize": "pad"` layout model, whereas
+Kibana uses `fit`. Remove all `autosize`, `width`, and `height`
+values. See link:#sizing-and-positioning[sizing and positioning] below.
+
+[[vega-additional-configuration-options]]
+==== Additional configuration options
+
+These options are specific to the Kibana.
+
+[source,yaml]
+----
+{
+  config: {
+    kibana: {
+      // Placement of the Vega-defined signal bindings.
+      // Can be `left`, `right`, `top`, or `bottom` (default).
+      controlsLocation: top
+      // Can be `vertical` or `horizontal` (default).
+      controlsDirection: vertical
+      // If true, hides most of Vega and VegaLite warnings
+      hideWarnings: true
+      // Vega renderer to use: `svg` or `canvas` (default)
+      renderer: canvas
+    }
+  }
+  /* the rest of Vega code */
+}
+----
+
+[[vega-sizing-and-positioning]]
+==== Sizing and positioning
+
+[[vega-and-vegalite]]
+Vega and VegaLite
+
+By default, Kibana Vega graphs will use
+`autosize = { type: 'fit', contains: 'padding' }` layout model for Vega
+and VegaLite graphs. The `fit` model uses all available space, ignores
+`width` and `height` values, but respects the padding values. You may
+override this behaviour by specifying a different `autosize` value.

package.json

@@ -37,7 +37,8 @@
     "Nicolás Bevacqua <[email protected]>",
     "Shelby Sturgis <[email protected]>",
     "Spencer Alger <[email protected]>",
-    "Tim Sullivan <[email protected]>"
+    "Tim Sullivan <[email protected]>",
+    "Yuri Astrakhan <[email protected]>"
   ],
   "scripts": {
     "test": "grunt test",
@@ -111,6 +112,7 @@
     "check-hash": "1.0.1",
     "color": "1.0.3",
     "commander": "2.8.1",
+    "compare-versions": "3.1.0",
     "css-loader": "0.28.7",
     "d3": "3.5.6",
     "d3-cloud": "1.2.1",
@@ -132,6 +134,7 @@
     "h2o2": "5.1.1",
     "handlebars": "4.0.5",
     "hapi": "14.2.0",
+    "hjson": "3.1.0",
     "http-proxy-agent": "1.0.0",
     "imports-loader": "0.7.1",
     "inert": "4.0.2",
@@ -140,6 +143,7 @@
     "joi": "10.4.1",
     "jquery": "3.2.1",
     "js-yaml": "3.4.1",
+    "json-stringify-pretty-compact": "1.0.4",
     "json-stringify-safe": "5.0.1",
     "jstimezonedetect": "1.0.5",
     "leaflet": "1.0.3",
@@ -207,6 +211,9 @@
     "url-loader": "0.5.9",
     "uuid": "3.0.1",
     "validate-npm-package-name": "2.2.2",
+    "vega": "3.0.8",
+    "vega-lite": "2.0.3",
+    "vega-schema-url-parser": "1.0.0",
     "vision": "4.1.0",
     "webpack": "3.6.0",
     "webpack-merge": "4.1.0",

src/core_plugins/tests_bundle/tests_entry_template.js

@@ -30,7 +30,11 @@ window.__KBN__ = {
     },
     mapConfig: {
       manifestServiceUrl: 'https://staging-dot-catalogue-dot-elastic-layer.appspot.com/v1/manifest'
-    }
+    },
+    vegaConfig: {
+      enabled: true,
+      enableExternalUrls: true
+    },
   },
   uiSettings: {
     defaults: ${JSON.stringify(defaultUiSettings, null, 2).split('\n').join('\n    ')},

src/core_plugins/vega/index.js

@@ -0,0 +1,15 @@
+export default kibana => new kibana.Plugin({
+  id: 'vega',
+  require: ['elasticsearch'],
+
+  uiExports: {
+    visTypes: ['plugins/vega/vega_type'],
+    injectDefaultVars: server => ({ vegaConfig: server.config().get('vega') }),
+  },
+
+  config: (Joi) => Joi.object({
+    enabled: Joi.boolean().default(true),
+    enableExternalUrls: Joi.boolean().default(false)
+  }).default(),
+
+});

src/core_plugins/vega/package.json

@@ -0,0 +1,6 @@
+{
+  "author": "Yuri Astrakhan<[email protected]>",
+  "name": "vega",
+  "version": "kibana"
+}
+