Filters as a native dashboard (v2) construct

[mistercrunch]

Making a list of ideas around how dashboard filters should work in dashboard v2.

Scope awareness. The filter affect only the charts and elements in its scope. Existing scopes should include:

• global (above tab-line)
• tab
• section
• more granular?

Maybe hovering the filter highlights the charts affected by the filter.

Per-filter configuration:

• type: select - radio - ? (radio probably not critical / MVP)
• multiple or single select
• sort order (based on metric / reverse / alphabetical)
• limit?
• show background bar (histogram)
• cascading-options (wether higher level filter should affect the values in the dropdown), maybe a whitelist blacklist approach?

Making it clear on charts that are filtered, maybe a filter icon that highlight which control are affecting the chart when hovered?

More input is welcomed.

[villebro]

Jinja templating filters, i.e. changes in the filter affect values in the template params. We currently have a few use cases where templates seem like a good solution for producing many related but still slightly different charts while using fewer data sources (currently working on a feature to enable overriding the datasource template params in the chart). This would probably need some sort of schema to be able to generate the filter, so probably not something for MVP, but potentially down the road.

[ahsanshah]

Some additional ideas on this:

• Any chance for Filtered MultiSelect with Selections like this: http://insin.github.io/react-filtered-multiselect/. This would be great when you have larger lists and can easily search on selected (right) and unselected (left). Not sure if this is a different viz or can be incorporated to existing viz

• Default Capabilities at filter level? First/Last Value by whatever Sort Order (Metric, Alphabetical).

Would any of these be viable?

[fordhamedu]

Has single select in Table view functioning as a “filter on” action been given any further thought? Similar to Tableau in some ways

[mistercrunch]

@fordhamedu there's clearly a need for better constructs around visualizations being able to post filter actions to the dashboard (like the Table viz can currently do). Not sure how those should be scoped (should it affect the whole dashbaord? the section? the tab?)

Maybe really good dashboard-native filtering removes some of the need for that.

[mistercrunch]

@fordhamedu there's clearly a need for better constructs around visualizations being able to post filter actions to the dashboard (like the Table viz can currently do). Not sure how those should be scoped (should it affect the whole dashbaord? the section? the tab?)

Maybe really good dashboard-native filtering removes some of the need for that.

[villebro]

On-the-fly grouping. I often find that my filters affect how I want to group my charts. If, for instance I filter by region I might want to se a chart grouped by product, but if I then change my filter to product, I want to see that same chart grouped by region. Etc. So having a groupby element which can be made to affect specific charts would be very valuable.

[fordhamedu]

the process of “select then deselect then select another” to isolate and explore the connected data in multiple charts within one dashboard can be counterintuitive to most users. Especially within longer tables it’s an easy oversight to forget the last deselect, and then confusing the data that’s displayed. A optional single select and optional filter on would enable a user to easily switch between aggregate and isolated selections and passing variables through markdown, iframe, and any url Parmaters On a single Dashboard. That would be awesome

[JamshedRahman]

In addition to showing background bar (histogram), would it be helpful to include statistics (e.g. % count, raw count) for each value? This way the user can get an accurate measure which may not be apparent from histogram? Or maybe the stats show up when you hover over the histogram bar.

[JamshedRahman]

We have some work in progress that would enable context-aware filtering for multiple filter boxes in dashboard / explore view. I feel that could be useful here. Adding @emlittleworth

[emlittleworth]

Hey everyone, as @JamshedRahman mentioned at Twitter we have work in progress for cascading (or context-aware) filtering.

Another project we have coming up that is relevant here is cross-filtering. We brought this up at the Superset meeting on 6/29. We haven't yet decided on design but are hoping it'd have similar functionality to this example: https://dc-js.github.io/dc.js/. For example, when you select "Loss" in the "Days by Gain/Loss" chart, the other charts filter using that value as well.

[mistercrunch]

@emlittleworth curious whether you're thinking client-side cross filtering or more of a "filter events emitted from within charts"?

The first one would require massive changes, I doubt whether it's actually reasonable or even possible to do this.

The second is totally doable, and the Table visualization can actually do this today if you turn on the feature.

[emlittleworth]

Definitely filter events emitted from within charts. This seems reasonable since we already have a notion of dashboard filters through the filter box. This feature would enable other visualizations to trigger dashboard-wide filtering. Thanks, I'll have to look into the Table visualization.

[mistercrunch]

Here's how it's done in the Table visualization:
https://github.com/apache/incubator-superset/blob/master/superset/assets/src/visualizations/table.js#L155-L159

It's more or less just a matter of hooking filter events up in other visualizations, though it's a bit more complicated than that. Beyond emitting the filtering event, it'd be nice to have visual components (tooltip, context menu, ...) that would make it clear what is about to happen when you click and present multiple options where necessary. While the table component knows about the dimensional context (we know one cell correspond to a single dimension value) it's easier there. A single bar in a bar chart may represent 2 dimension members and may have to emit 2 filters.

[emlittleworth]

Oh yeah good point about multiple dimensions. And I agree about the visual components as well. Another thing to keep in mind there is how we should display charts that cannot be filtered because they don't contain the filter dimension.

[mistercrunch]

Ideally the dashboard app should make it very clear which filters are active and which charts they affect. Maybe hovering an active filter components would highlight the targets, or hovering a filtered chart would highlight the related "source" filters.

[emlittleworth]

Yeah that sounds good. Can you elaborate your thinking on cascading filters? What do you mean by "maybe a whitelist blacklist approach"?

[jaylindquist]

@mistercrunch

We would like to add a proposal for the native dashboard filter as outlined in the first post. We are looking for feedback and approval before implementing.

Dashboard changes

• Global filters will be the focus of the MVP
  • Tab and section filters would work similarly but will not be included in the initial implementation. The initial focus will be to replace the existing Filter Box chart with a native, global dashboard filter
• The filter component is drag & drop from the insert panel, similar to tabs
  • Global filters go above dashboard-level tabs and below the dashboard title (see images below for an example)
  • Tab filters go below tabs
  • Section filters go at the top of the section
• New filters can be added when the dashboard is in "Edit Mode" via a button that appears after the list of filters
  • Individual filters are added based on datasource / column name
  • Clicking the New Filter button will open a modal with "Data Source" and "Column" options to select the filter column
  • If the column selected is a time column, the 'Since' / 'Until' filters will be added
• Filter controls for the dashboard are initially collapsed and can be expanded / collapsed as needed (see images below for an example)
• Additional options such as timegrain and instant filtering are added via the configuration cog

Chart changes

• Filter box will be removed as a chart type
• Filter boxes that are currently included in dashboards will need to be removed from the dashboard and options will be added as global filter options
• All existing filter box charts are deleted (not sure what else can be done with them since they can't be added to dashboards anymore)

URL changes

• The preselect_filters query parameter can still be used to fill in filters via a link. It is currently a map which includes the Filter Box ID. The format will need to change into a map of column name to arrays instead
  • Old format

  {
      "414": {
          "column1": [
              "value1",
              "value2"
          ]
      },
      "12": {
          "column2": [
              "value3"
          ]
      }
  }

  • New format

  {
    "column1": [
        "value1",
         "value2"
    ],
    "column2": [
        "value3"
    ]
  }

  • This change can be made backwards compatible by collecting the filters in the old format, ignoring the ID, and applying the filter values based on the column name

Dashboard mockups

Collapsed filters

Opened filters

New filter dialog

[graceguo-supercat]

hi @jaylindquist i like your proposal.
Can we have a more generic time range filter, which is used for setting time range for charts in scope (global, single tab level or single section)? It doesn't associated with any datasource, but when apply this time filter to each chart, probably need some smart way to bind to the right column of different chart.

[jaylindquist]

@graceguo-supercat Today superset has a "Time Column" for filters that it uses to manage which column to filter on.

I'm not sure how a generic time filter would work. For instance, a table of events that have a start_date column and end_date column. If a user wanted to filter on all events starting in January, they wouldn't want the filter to apply to the end_date column as well.

They would need to specify which column(s) the time range should apply to.

[jaylindquist]

@mistercrunch @graceguo-supercat Just pinging to see if there is any other feedback before we start planning our implementation

[mistercrunch]

Deprecating filter_box may be tricky as you'll need a [perhaps complex] db migration script, and that may leave the layout of dashboards messed up (users would find an empty hole).

Personally I'm thinking we could just put the viz type on life support and add a "deprecated" stamp on the thumbnail png, or making it inaccessible from the viz type switcher modal. Eventually could encourage dashboard owners to remove those from their dashboards, maybe some alert when in edit mode.

I think the add/edit filter modal needs more options: whether it should be based on a metric, sorting specification, allow_multi_choices.

[graceguo-supercat]

I collected following feature requests from open source community, and from Airbnb usage:

• Filter component or filter_box from explore view? Given @mistercrunch comments above, I assume existed filter_box viz type will keep working as it, and will introduce new filter component lived inside dashboard. So there will be two types of filters working together.

• Scoped filter

  • Filter should be visible (scroll up/down is ok, but not extra tab switch) when it is applicable to any charts.
  • Top level filters:
    Filter is placed in dashboard without tab, or at top section of dashboard with row-level tabs. So that filters are always visible in the dashboard.
    Top level filters will apply to all charts in the dashboard.
  • Tab level filters:
    Tab level filters are placed inside a Tab component. As long as the tab component is visible, filters will apply to all charts in the same tab.

• Easy UI for user to see which charts are affected by which filter, and easy add immune from the filter.

• Easy UI to group multiple filters, and allow user define cascade filters.
  Current filters are filter_box that generated from explore view, so each of filter_box (can have multiple dropdowns) contains one slice of constraint, whereas dashboard may have multiple filter_boxes to add many constrains. If this is the case, i see our users always update one filter_box in dashboard, apply it, wait all charts in dashboard finished loading, then update another filter_box, wait another round of loading...It is very frustrating in daily use.
  
  So I am thinking dashboard should allow user group a few filter_boxes(and native filter components) together, in which user can update each filter but don't trigger dashboard level loading events. In the group user can also make cascade filters (where one filter react on other filters's selected values). And in the end user can apply all selected values altogether in 1 dashboard level load. This will greatly improve the dashboard filters usability.

• After user update filter and trigger the dashboard level refresh, can we have UI indicator showing filter is applying or done? Some of our dashboard is crazy big so it's hard to scroll back and forward to check if every chart is ready 😓

• Keep filters state in URL parameter [Filters are not kept when exploring from dashboard #5374]
  If user applied filter in dashboard, can we update dashboard's url parameter (or anywhere as a state of history), like url for explore view? So that when user refresh dashboard, the previously applied filters are still be memorized.
  Dashboard should not automatically save filter parameter unless user saved them as default_filter metadata.

• Coordinate Visual Correlation feature. Discussed in another thread, @JamshedRahman want to introduce a new feature that allow one chart to publish events to dashboard level, and all other charts in the dashboard that are listening to the events can change/react accordingly. These 2 features will implemented independently, but i just feel there might be some common logic to share.

These are all proposals. I would like to leave @xtinec who implements this feature to decide final list. Thank you!!

[AndreLesa]

Is there somewhere to track the progress of these changes?

[jaylindquist]

I apologize for the lack of updates on this. Our team hit a series of emergencies that took priority over the filter changes. We've only now been able to start the work.

Changes will be going into my forked at https://github.com/jaylindquist/incubator-superset/tree/feature/native-filter

[jaylindquist]

Due to our changing priorities, we have been unable to complete the dashboard level filters feature. You can find our in progress work on my feature branch (https://github.com/jaylindquist/incubator-superset/tree/feature/native-filter) and I can create a PR if you are interested in keeping what we've been able to start.

Completed tasks:

• Add a drag-and-drop filters container at the top of dashboards
• Add filters to the filter container (filters are added, but configuration is hard coded)
• Remove filters from container
• Remove filters container from dashboard
• Update Dashboard short links to create preselect_filters parameter without chart IDs. URL short links are now generated as { "column1": ["val1"], "column2": ["val2", "val3"] }
• Update preselect_filters parameter to support filters without chart IDs while being backwards compatible. preselect_filters now supports the above format as well as the existing format { "chartId1": { "column1": ["val1"]}, "chartId2": { "column2": ["val2", "val3"]}}

The following shows adding filters to an existing dashboard:

There is still work to be done:

• Creating a modal to add new filters
• Manage date filters for each datasource
• Global filter configuration (for example, instant filtering)
• Saving selected filter options when the dashboard is saved

If you would like me to create a PR for this work, let me know. I apologize for not being able to complete the feature, but other tasks took priority for us.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[D-Mahajan]

Any update on Cascading filters?

[sharpyfox]

Hello. Do you have any ETA about native filters?

[zpanderson]

Also curious about this. Doesn't look like any work on Jay's branch since 2018.

[tvorogme]

=(

[rcarboni]

Piling on for the cascading filter logic (i.e. in a multiple-filter widget, only allow combinations that exist as a row in the filter datasource; and some precedence logic, like top-to-bottom).

You can achieve the above by creating each filter individually (and setting the precedence logic in the slice immunity configs), but then you lose the option of a single refresh button to prevent all the charts from refreshing when you're in the middle of updating filter selections. So conversely to cascading logic, it would be good to have:

• a "refresh" widget that could gate multiple filters and only publish them when the button is pressed
• some visual queue on each of the filters that is awaiting a refresh (i.e. filters that have changed from their published value, and will be updated once the button is pushed

Lastly, a "refresh" button is also good aesthetically if you have a lot of filters, since you have freedom in placing them, vs. a single widget where they all have to be stacked vertically.

[rcarboni]

One more thing for filters, it would be great if I could specify 'value' and (optional) 'label' fields, so that I can use more informative names to display in the filter, and pass the more functional value back to a query, e.g.

Aggregation Filter:

• label: "Agg by Sum", value: "SUM("
• label: "Agg by Average", value: "AVG("

[avinasht1]

Would love this much needed feature.

[EBoisseauSierra]

The last commit on the branch mentionned by @mistercrunch is about one year old.

Should we consider this feature request dead? Are there alternative plans considered?

[kalimuthu123]

the global idea filter request is good we will move forward to nxt levels

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

[thunderbird57]

We still need this... I recently started using Superset. I loved the tool but filtering constraint is making it hard to sell it to my team who will be working on it.