[Canvas] XY_VIS

[Kuznietsov]

Adapt xy chart from vis_types/xy to be reused first at Canvas.

Linked issue: #101377

Description of the problem.

Currently, xy_vis function accepts further arguments, which are making this expression not acceptable to be reused somewhere else with some other data source:

---

Full description of the fn interface:

Input:

---

{ 
  type?: string = "";
  chartType?: 'line' | 'area' | 'histogram'; 
  addTimeMarker?: boolean;
  addLegend?: boolean;
  addTooltip?: boolean;
  legendPosition?: 'top' | 'left' | 'right' | 'bottom';
  orderBucketsBySum?: boolean;
  radiusRatio?: number;
  gridCategoryLines?: boolean;
  gridValueAxis?: string;
  isVislibVis?: boolean;
  detailedTooltip?: boolean;
  fittingFunction?: string;
  palette?: string;
  fillOpacity?: number;
  labels?: Label; // `label` function
  truncateLegend?: boolean;
  maxLegendLines?: number;

  times?: Array<{
    time: string;
    class?: string;
    color?: string;
    opacity?: number;
    width?: number;
  }>; // `time_marker` function

  seriesParams?: Array<{
    label: string;
    id: string;
    drawLinesBetweenPoints?: boolean;
    interpolate?: string;
    show: boolean;
    lineWidth?: number;
    mode?: string;
    showCircles?: boolean;
    circlesRadius?: number;
    type?: string;
    valueAxis?: string;
  }>; // `series_param` function

  valueAxes?: Array<{
     name: string;
     axisParams: {
        id: string;
        show: boolean;
        position: string;
        type: string;
        title?: string;
        scale: {
          boundsMargin?: number | string;
          defaultYExtents?: boolean;
          setYExtents?: boolean;
          max?: number  | null;
          min?: number | null;
          mode?: string;
          type: string;
        }; // `vis_scale` function
        labels: Label; // `label` function
      };
  }>; // `value_axis` function

  categoryAxes?: Array<{
    id: string;
    show: boolean;
    position: string;
    type: string;
    title: string;
    scale: {
      boundsMargin: number | string;
      defaultYExtents: boolean;
      setYExtents: boolean;
      max: number  | null;
      min: number | null;
      mode: string;
      type: string;
    }; // `vis_scale` function
    labels: Label; // `label` function
  }>; // `category_axis` function
  
  thresholdLine?: {
    show: boolean;
    value: number | null;
    width: number | null;
    style: string;
    color: string;
  }; // `threshold_line` function;

  xDimension?: {    
    visDimension: {
        accessor?: string | number;
        format?: string = 'string';     
        formatParams?: string = "{}";
    }; // `vis_dimension` function
    label?: string;
    aggType?: string;
    params?: string = "{}";
  } | null; // `xy_dimension` function

  yDimension?: Array<{    
    visDimension: {
        accessor?: string | number;
        format?: string = 'string';     
        formatParams?: string = "{}";
    }; // `vis_dimension` function
    label?: string;
    aggType?: string;
    params?: string = "{}";
  }> | null; // `xy_dimension` function

  zDimension?: Array<{    
    visDimension: {
        accessor?: string | number;
        format?: string = 'string';     
        formatParams?: string = "{}";
    }; // `vis_dimension` function
    label?: string;
    aggType?: string;
    params?: string = "{}";
  }> | null; // `xy_dimension` function
  widthDimension?: Array<{
    visDimension: {
        accessor?: string | number;
        format?: string = 'string';     
        formatParams?: string = "{}";
    }; // `vis_dimension` function
    label?: string;
    aggType?: string;
    params?: string = "{}";
  }> | null; // `xy_dimension` function;

  seriesDimension?: Array<{    
    visDimension: {
        accessor?: string | number;
        format?: string = 'string';     
        formatParams?: string = "{}";
    }; // `vis_dimension` function
    label?: string;
    aggType?: string;
    params?: string = "{}";
  }> | null; // `xy_dimension` function

  splitRowDimension?: Array<{    
    visDimension: {
        accessor?: string | number;
        format?: string = 'string';     
        formatParams?: string = "{}";
    }; // `vis_dimension` function
    label?: string;
    aggType?: string;
    params?: string = "{}";
  }> | null; // `xy_dimension` function

  splitColumnDimension?: Array<{    
    visDimension: {
        accessor?: string | number;
        format?: string = 'string';     
        formatParams?: string = "{}";
    }; // `vis_dimension` function
    label?: string;
    aggType?: string;
    params?: string = "{}";
  }> | null; // `xy_dimension` function;
}

Output:

---

{ 
  type?: 'line' | 'area' | 'histogram'; 
  addTimeMarker?: boolean;
  addLegend?: boolean;
  addTooltip?: boolean;
  legendPosition?: 'top' | 'left' | 'right' | 'bottom';
  orderBucketsBySum?: boolean;
  radiusRatio?: number;
  grid: { 
    categoryLines: boolean;
    valueAxis: string;
  };
  isVislibVis?: boolean;
  detailedTooltip?: boolean;
  fittingFunction?: string;
  palette: {
    type: 'palette';
    name?: string;
  };
  fillOpacity?: number;
  labels?: Label; // `label` function
  truncateLegend?: boolean;
  maxLegendLines?: number;

  /* This functions described below in the text. */
  times?: Array<{
    time: string;
    class?: string;
    color?: string;
    opacity?: number;
    width?: number;
  }>; // `time_marker` function

  seriesParams?: Array<{
    type: 'line' | 'area' | 'histogram';
    data: { label: string; id: string };
    drawLinesBetweenPoints?: boolean;
    interpolate?: 'linear' | 'cardinal' | 'step-after';
    lineWidth?: number;
    mode?:   'normal' | 'stacked';
    show: boolean;
    showCircles?: boolean;
    circlesRadius?: number;
    valueAxis?: string;
  }>; // `series_param` function

  valueAxes?: Array<{
    name: string;
    id: string;
    show: boolean;
    position: 'top' | 'left' | 'right' | 'bottom';
    type: 'value' | 'category';
    title?: {
      text?: string;
    };
    labels?: Array<{
      color?: string;
      filter?: boolean;
      overwriteColor?: boolean;
      rotate?: LabelRotation;
      show?: boolean;
      truncate?: number | null;
    }>;
    scale?: {
      boundsMargin?: number | '';
      defaultYExtents?: boolean;
      max?: number | null;
      min?: number | null;
      mode?: AxisMode;
      setYExtents?: boolean;
      type: 'linear' | 'log' | 'square root';
    };
  }>; // `value_axis` function

  categoryAxes?: Array<{
      id: string;
      show: boolean;
      position: 'top' | 'left' | 'right' | 'bottom';
      type: 'value' | 'category';
      title: {
        text?: string;
      },
      labels: {
        color?: string;
        filter?: boolean;
        overwriteColor?: boolean;
        rotate?: LabelRotation;
        show?: boolean;
        truncate?: number | null;
      },
      scale: {
        boundsMargin?: number | '';
        defaultYExtents?: boolean;
        max?: number | null;
        min?: number | null;
        mode?: 'normal' | 'percentage' | 'wiggle' | 'silhouette';
        setYExtents?: boolean;
        type: 'linear' | 'log' | 'square root';
      }
  }>; // `category_axis` function
  
  thresholdLine?: {
    show: boolean;
    value: number | null;
    width: number | null;
    style: string;
    color: string;
  }; // `threshold_line` function;

  dimensions: {
    x?: {
      label?: string;
      aggType?: string;
      params?: {
        date: boolean;
        interval: number;
        intervalESValue: number;
        intervalESUnit: string;
        format: string;
        bounds?: {
          min: string | number;
          max: string | number;
        };
      } 
      | {
        interval: number;
      } 
      | {
         defaultValue: string;
      } 
      | {} = "{}";
      accessor: number; // need to fix accessor type here to enable support of string accessor
      format: SerializedFieldFormat;
    }; // `xy_dimension` function

    y?: Array<{
      label?: string;
      aggType?: string;
      params?: {
        date: boolean;
        interval: number;
        intervalESValue: number;
        intervalESUnit: string;
        format: string;
        bounds?: {
          min: string | number;
          max: string | number;
        };
      } 
      | {
        interval: number;
      } 
      | {
         defaultValue: string;
      } 
      | {} = "{}";
      accessor: number; // need to fix accessor type here to enable support of string accessor
      format: SerializedFieldFormat;
    }>; // `xy_dimension` function 

    z?: Array<{
      label?: string;
      aggType?: string;
      params?: {
        date: boolean;
        interval: number;
        intervalESValue: number;
        intervalESUnit: string;
        format: string;
        bounds?: {
          min: string | number;
          max: string | number;
        };
      } 
      | {
        interval: number;
      } 
      | {
         defaultValue: string;
      } 
      | {} = "{}";
      accessor: number; // need to fix accessor type here to enable support of string accessor
      format: SerializedFieldFormat;
    }>; // `xy_dimension` function 

    width?: Array<{
      label?: string;
      aggType?: string;
      params?: {
        date: boolean;
        interval: number;
        intervalESValue: number;
        intervalESUnit: string;
        format: string;
        bounds?: {
          min: string | number;
          max: string | number;
        };
      } 
      | {
        interval: number;
      } 
      | {
         defaultValue: string;
      } 
      | {} = "{}";
      accessor: number; // need to fix accessor type here to enable support of string accessor
      format: SerializedFieldFormat;
    }>; // `xy_dimension` function 

    series?: Array<{
      label?: string;
      aggType?: string;
      params?: {
        date: boolean;
        interval: number;
        intervalESValue: number;
        intervalESUnit: string;
        format: string;
        bounds?: {
          min: string | number;
          max: string | number;
        };
      } 
      | {
        interval: number;
      } 
      | {
         defaultValue: string;
      } 
      | {} = "{}";
      accessor: number; // need to fix accessor type here to enable support of string accessor
      format: SerializedFieldFormat;
    }>; // `xy_dimension` function 

    splitRow?: Array<{
      label?: string;
      aggType?: string;
      params?: {
        date: boolean;
        interval: number;
        intervalESValue: number;
        intervalESUnit: string;
        format: string;
        bounds?: {
          min: string | number;
          max: string | number;
        };
      } 
      | {
        interval: number;
      } 
      | {
         defaultValue: string;
      } 
      | {} = "{}";
      accessor: number; // need to fix accessor type here to enable support of string accessor
      format: SerializedFieldFormat;
    }>; // `xy_dimension` function 

    splitColumn?: Array<{
      label?: string;
      aggType?: string;
      params?: {
        date: boolean;
        interval: number;
        intervalESValue: number;
        intervalESUnit: string;
        format: string;
        bounds?: {
          min: string | number;
          max: string | number;
        };
      } 
      | {
        interval: number;
      } 
      | {
         defaultValue: string;
      } 
      | {} = "{}";
      accessor: number; // need to fix accessor type here to enable support of string accessor
      format: SerializedFieldFormat;
    }>; // `xy_dimension` function 
  }
}

---

xy_dimension

input arguments:

{
    visDimension: VisDimension; // `vis_dimension` function
    label: string;
    aggType: string;
    params: string
}

output arguments:

{
    label: string;
    aggType: string;
    params: DateHistogramParams | HistogramParams | FakeParams | {};
    accessor: number | DatatableColumn;
    format: SerializedFieldFormat;
}

interface DateHistogramParams {
  date: boolean;
  interval: number;
  intervalESValue: number;
  intervalESUnit: string;
  format: string;
  bounds?: {
    min: string | number;
    max: string | number;
  };
}

interface HistogramParams {
  interval: number;
}

interface FakeParams {
  defaultValue: string;
}

---

The biggest issue at this function:

I see the biggest trouble is the format of xy_dimension function. Here is my point:

This function accepts fields, related to es aggregations.
List of fields:

    aggType: string;
    params: DateHistogramParams | HistogramParams | FakeParams | {};
    params.intervalESValue: number;
    params.intervalESUnit: string;
    params.bounds?: {
      min: string | number;
      max: string | number;
    };
   params.interval: number;

I think, xy_vis function should avoid taking arguments from kibana_context and making decisions, based on that params, instead, it should receive necessary flags and metrics, defined on the top level. It should only visualize the given chart without special logic.

---

I propose:

• convert x, y, z, series, splitColumn and splitRow from xy_dimension to updated form of xy_dimension interface, without aggType and params.* specific es_aggs fields.
• add function for xDomain and adjustedXDomain which would return { min?: number, max?: number, interval?: number }. I think it will be a good way of supporting the special logic of xDomain with a specially defined interval. This param is optional, based on the xy chart behavior. The decision, which interval to use, will be made on the level of toExpressionAst.
• transform the logic of getConfig method and further operations. It is very complicated and, I believe, should be moved to xy_vis function (or renderer, not sure) body.

---

Steps to perform:

• Break the connection between chart and es aggregation attributes.
• Move all the logic of transforming parameters from chart to xy_vis function (or renderer, not sure).
• Think about simplifying the API of the expression and compatibility with other teams' usage.

---

What is the purpose of such changes?

• For now, this visualization is strongly connected to the logic of aggregation result and parameters, so, cannot be used at other plugins.
• Canvas is providing the possibility to fetch data not only by es_aggs but also by demodata, es_docs, etc, so, we should break the connection between displaying and the way of fetching data.
• Any function at the expressions don't have access to kibana_context and to data plugin, except the top-level function, as you can mention, so they cannot provide arguments, defined at aggregation execution context.
• We need to provide flexibility in the configuration of the chart view, not connected to the way of fetching data.
• This function's API needs to be simplified because it is very complicated to understand and use (please, keep in mind, it will be used at the Expression Editor at Canvas.

---

Steps to complete:

• Step 1. Remove all specific logic, related to aggType.
• Step 2. Move xDomain to the expression function and update types.
• Step 3. Move XY functions and renderer from vis_types to chart_expressions.
• Step 4. Update handlers.
• Step 5. Simplify XY function API. #112935
• Step 6. Add the ability to add more chart types.
• Step 7. Add tests to all functions at XY.
• Step 8. Add tests to utils at XY.
• Step 9. Add tests to all charts at XY.
• Step 10. Integrate XY to Canvas.
• Step 11. Add arguments to XY expression at Canvas.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[elasticmachine]

Pinging @elastic/datavis (Feature:ElasticCharts)

[flash1293]

Chatted about this with @Kunzetsov and it's planned to take Lens into account in step 4 of the plan in the description of this issue.

Two things we should probably consider here:

• Simplify XY function API is not really a good description - to be able to handle all Lens features, the API will actually become more complex (multiple and dynamic thresholds, multiple layers (each with its own palette), auto-axis assignments to just name the one that come to mind). We should start figuring out a plan how to do this soon because it seems like we will get to this step in the near term. The API changes won't be trivial and shouldn't be considered ad-hoc after all other steps are completed
• We should add a step to switch to the new unified renderer for Lens (doesn't have to happen right away)

fyi @stratoula