[Lens] [Discuss] Dimension workflow

[cchaos]

There have been many conversations around how the configuration panel should handle editing a dimension (field, aggregation, and aggregation options).

Here is my proposal as talked it over a bit with @flash1293

All options in a single popover, with all aggregations available (similar to POC)

Allow users to explore all types of aggregations for the dimension type but simply show an error as to why a selected field does not work with the selected aggregation. Do not automatically change the field to one that is the correct field type. And do not remove options that don't work with the selected field.

Pros:

• This allows for greater exploration of all possible combinations of fields and aggregations
• Keeps the config panel simplified
• The config panel acts as a human-readable summary of their dimension and aligns the aligns the order of the summary with the order of operations in the popover:

Cons:

• May cause users to end up in an error state more often
  • However, this is somewhat mitigated by the ease of switching to other aggregations and the application being quick to respond (re-builds the query each time)
• Users don't think in terms of aggregations, they think in terms of their data

  • Users def are looking for "getting to the field I want" and not the aggregation 🙂 That was one of the main problems with the old visualize editor, that users needed to go over aggregations and we figured that out (in a lot of trainings, discuss forums, etc.) that that's basically not how users think at all. They care about their data, and they also would never want to switch to an aggregation that's not available for that data -@timroes via Slack

  • My comment to that is, in order to get to this popover state, they have already chosen their field (drag and dropped to the dimension). The field selector is there if they realize they chose the wrong field. This means that their main reason for getting to this popover is to change the aggregation and/or aggregation options.

There are essentially 2 ways to create a dimension:

And they are both field-first:

1. Drag a field from the fields panel
2. Click on the config panel to initiate, in which case we can still force them to choose a field first:

Another way to layout the panel

Would be to put the field selection up top, then the aggregation and options below. However, we lose the 3rd point made in the pros section:

---

IF (and it's a big IF) we must limit the aggregation list based on field selection

Then we're basically back to the current vis editor UI (swap aggregation and field) where it's a top down approach.

Which doesn't align with the human-readable summary and causes more clicks to change the aggregation and options.

---

cc @wylieconlon @chrisdavies @AlonaNadler

[wylieconlon]

This is pretty similar to what we've talked about, but I'd like to clarify it in using a specific and very limited example.

We're not assuming that our dragging is accessible, so to have an accessible way of editing dimensions we need it to be possible to edit a dimension without a prior field selection. So in this example, we are starting with a blank state. Let's assume that we are only allowing aggregations. The user:

1. Clicks on the config panel for the X axis of a vertical bar chart
2. In this example the user will choose a field first- could be forced or not
3. There are two questions: Which fields can be placed on the X axis, and which functions can be used on those fields?

Our bar chart should require that the X axis is bucketed by date histogram or terms. So in the field dropdown, I would only expect to see string or date fields. No other fields would ever work on the X axis, so I propose hiding them. Do you agree?

Let's say the user selects a string field like "category" as their X axis. Using our suggestions, we would automatically select the "Top Values" function with the default of the Top 5. In the list of other functions, there are no other possible functions- it's not possible to do a date histogram of a string, or to use a numeric function like Cardinality on the X axis.

If I understand your proposal correctly, you are proposing that if the user changes their mind after seeing Top 5 Values of Category, there should be clickable options for changing either the field or the function. You are also proposing that there is an invalid state, so that in this example the user would be able to select "Date Histogram" and be shown a warning message saying "Date Histogram only works with date fields, and this is a string field." This could work, but I'm not sure introducing the invalid state improves the UX of this example. Is there another option?

This is a really limited example because the only possible selections are Date Histogram or Terms with a small number of fields, but I think we can describe all the possible functions as fitting into one of these four categories:

a. Functions that are valid based on the current field and dimension
b. Functions that are valid on the dimension, but never usable with the current field. For example, the Date Histogram function is valid on X but not with a string field.
c. Functions that are valid on the dimension, but not usable with other dimensions. For example, if we support both raw and aggregated functions, you can't mix them across dimensions.
d. Functions that will never be allowed in the current dimension because of visualization rules, such as a non-bucketed function on the X axis.

The biggest question that we haven't yet decided on is whether to introduce an invalid state where the function+field+other dimensions combination could be invalid, or whether to prevent users from entering those invalid states. I think the possibilities are:

1. Allow users to select functions that don't work with the current field (case B above), and in this case automatically change the field
2. Allow users to select functions that don't work with the current field (case B above), and enter an invalid state
3. Prevent users from selecting functions that don't work with the current field (case B above), and require the user to change the field

My preference is for option 3, which you can see in action in #38863

Another question that you've raised is whether to allow users to select a function first. If we supported this, what would users expect the field selector options to be after selecting a function? Normally, the field selector would show all possible fields that can work with that dimension. Would users expect two different behaviors in the field selector based on context?

[cchaos]

Our bar chart should require that the X axis is bucketed by date histogram or terms. So in the field dropdown, I would only expect to see string or date fields. No other fields would ever work on the X axis, so I propose hiding them. Do you agree?

✅

If I understand your proposal correctly, you are proposing that if the user changes their mind after seeing Top 5 Values of Category, there should be clickable options for changing either the field or the function.

✅

You are also proposing that there is an invalid state, [...] This could work, but I'm not sure introducing the invalid state improves the UX of this example. Is there another option?

This invalid state is what I'm proposing because it tells the user why this field-aggregation combination does not work. By limiting their set based on their field does not inform the user why some option is missing. This is the only option I'm proposing at the moment.

c. Functions that are valid on the dimension, but not usable with other dimensions. For example, if we support both raw and aggregated functions, you can't mix them across dimensions.

Let's not worry about this case for now

1. Allow users to select functions that don't work with the current field (case B above), and in this case automatically change the field

This is what I'm trying to avoid. That's how the POC works and when users had Histogram of Carrier then switched to Date histogram it automatically changed the field to timestamp which they a) didn't notice happen and b) couldn't get back to their original field-aggregation combo

I'm advocating for option 2, you're advocating for option 3. This might be a good one to test with users.

Another question that you've raised is whether to allow users to select a function first.

I am not advocating for this.

[AlonaNadler]

Many of our users have hundreds of fields and drop downs that shows all fields will not serve them well, they will have too many options that can be too similar with long field name. In the field panel, we can treat these problems more gracefully allowing users to search, filter and see all the field names, types and possible values.

I think we will benefit more by presenting limited number of the most used aggregation based on the field type and the less common/advance aggregations can require an extra click, that way for regular users it is obvious which aggregations go well with their fields and for advance users another click will open more advanced options e.g. pipeline aggregation.

Allowing them to change the aggregation after adding the data to an aggregation that doesn't work with the field they already selected creates confusion more than it benefits IMO. Maybe we can start without it and if users want we will introduce it later but it also not such a priority ATM

[wylieconlon]

I want to make sure we don't lose this, as @cchaos summarized our latest agreement about how this behavior could work:

So I think the flow we decided on is the following (with everything happening in the popover):

1. Click add dimension
2. Opens popover with all fields and all aggregations available
   3a. Choose Field first, it sort the aggregations list to put the most appropriate at the top, auto-selecting the first
   3b. Choose Aggregation first, it will sort the fields list to put the most appropriate at the top, do not auto-select
   4a. If you select a field that doesn’t work with the current aggregation, it will re-sort the agg list and auto-select the new top aggregation
   4b. If you select an aggregation that doesn’t work with the current field, it will give you a little warning that the combo doesn’t work and to select a different field

I think this behavior should be able to solve all of the concerns raised with "magical" switching and with showing too many options. Because there are very few aggregations supported in the initial beta plan, we probably don't need to hide those aggregations.

[wylieconlon]

I believe we are happy with the current implementation of the dimension editor, and if not then it might be more useful to create a new issue for this. With this understanding I'm closing the issue, if there's disagreement feel free to reopen.