Visualize "field" is a required parameter

[stacey-gammon]

Happening in master, using an imported packetbeat dashboard (imported via packetbeat 5.0)

Investigating...

[stacey-gammon]

There is at least one issue in that the packetbeat import_dashboards script is creating a packetbeat index in kibana when that index doesn't actually exist. I don't know if there is anything we can do about this, since anyone can use the es api directly to inject an index into .kibana.

There may be an additional issue in that the fields that are being imported are missing their types.

I'm hoping this is simply an issue from importing 5.0 packet-beat dashboards into kibana master.

[stacey-gammon]

Once I started tracking data and refreshed the fields, type is assigned and the dashboards work as expected.

I suppose this could be considered working as expected, though it isn't a great user experience.

[stacey-gammon]

Re-opening, as we should first verify whether this was working in 5.1. If it was, then this is a regression and something broke.

If something did break, I am not sure yet if the bug is on the beats side or the kibana side.

**Update: Not a regression. This exists in 5.1.1

[tsg]

I think this is due to elastic/beats#3147 (i.e. a bug on the Packetbeat side).

[Hronom]

I'm also use manual adding of index-patterns, visualizations and dashboards into .kibana index, and I'm also met this problem.

So the problem is that you put wrong data in fields inside type index-pattern:

...
"index-pattern": {
        "properties": {
          "fieldFormatMap": {
            "type": "text"
          },
          "fields": {
            "type": "text"
          },
          "intervalName": {
            "type": "text"
          },
          "notExpandable": {
            "type": "boolean"
          },
          "sourceFilters": {
            "type": "text"
          },
          "timeFieldName": {
            "type": "text"
          },
          "title": {
            "type": "text"
          }
        }
      },
...

And for Kibana devs:
I'm think that some API for manual adding of index-patterns, visualizations, dashboards and embed links generation from external programs will be good and for me it's highly needed.

Or at least add some documentation about how and what need to add in .kibana index.

[stacey-gammon]

Note that another user is running into this error with 5.1.1 and metricbeat dashboards, but I can't reproduce it.

https://discuss.elastic.co/t/saved-field-parameter-is-now-invalid-please-select-a-new-field-visualize-field-is-a-required-parameter/70034/2

It also looks like the user who reported the issue with packetbeat dashboards is not able to resolve the issue, even after getting data and refreshing the field list (he actually notes that refreshing the field list caused the dashboard to subsequently fail).

https://discuss.elastic.co/t/visualize-field-is-a-required-parameter-error/69536/7

[jbudz]

Another case at https://discuss.elastic.co/t/5-1-1-discover-error/70940, on first glance not using beats data.

[weltenwort]

And another report on discuss with a very simple setup: https://discuss.elastic.co/t/saved-field-parameter-is-now-invalid-please-select-a-new-field/71044

[stacey-gammon]

In the last two cases the error is being thrown on the Discover tab, so it might be being triggered by something else. I think this error appears for a variety of reasons.

The reason it was happening on the visualize and dashboard tab is that a visualization was using a field that was not marked as searchable or aggregatable. I suspect something similar (perhaps yet another issue with the field_stats api not returning data), but I do wonder why it's being triggered on the discover tab.

[jakommo]

@stacey-gammon I could imagine that it happens for users that are upgrading to 5. E.g. they used a field that was valid in 4.x (before field stats were used) and now is not "valid" anymore.

[Bargs]

The field agg param is set as the default time field in the histogram at the top of the Discover results, so for users seeing this in Discover their timestamp field is probably marked as not-aggregatable.

[Bargs]

This likely won't be fixed by ES anytime soon: elastic/elasticsearch#22438

So we should come up with a workaround in Kibana because this is obviously affecting a lot of users in different ways.

@critix would you mind elaborating on your use case here? What's the scenario in which you need to build visualizations on fields with no data?

One possible solution: I'm thinking for fields that are missing from field_stats (in other words, missing data), we mark them with a "maybe" instead of true/false for searchable and aggregatable. For "maybe" fields we remove the early failure from the Field Agg Param. We could then display them with a note in the agg config editor's field dropdown saying something like "these fields may be aggregatable, use at your own risk".

[stacey-gammon]

Discussed in Mendit Monday and confirmed this is a problem and we want to prioritize a solution. Ideally the solution would come from the elasticsearch side and we'd like to push forward on improving the field stats API. @spalger is going to follow up on elastic/elasticsearch#22438.

[cehrig]

I'm facing the same issue since i added a nested field to a mapping:

Is there any workaround?

// EDIT: I can see/search documents again, by deleting and recreating Index Patterns (not Refreshing).

[Bargs]

@cehrig is you're field mapped as an actual nested type? If so, Kibana does not currently support nested aggregations #1084

[jacobweber]

Just ran into this when exporting Kibana objects from one 5.2 instance, and importing them into another one.

I was able to work around it. Once I had some valid data in my indexes, I went to Management > Index Patterns, and refreshed the field list. That marked some of the fields as "searchable", which weren't that way before. Then I deleted and re-imported my objects.

[guyboertje]

@stacey-gammon - FYI, I am also getting this problem on 5.4.0 when testing the logstash modules feature that we are writing now.

[epixa]

@LeeDr What's there to discuss with this? Seems like something we need to fix

[LeeDr]

@epixa Discuss how to fix it? I don't think anybody knows the solution yet. And/or a work-around if possible for people hitting it.

[Bargs]

The solution for most of these issues is to switch to the field_caps API: #11114

Aside from that, there are a couple scenarios where this error may occur and the user can fix it on their own:

• Index pattern was created before data was indexed. Solution: Refresh field list after indexing some data.
• An imported 4.x visualization was created using a field that is no longer aggregatable because Elasticsearch turned off fielddata by default in 5.0. Solution: Enable fielddata on the field or switch the visualization to a keyword version of the field.
• A migration from 2.x to 5.3.1 runs into _field_stats call on geo_point field broken after upgrading from 2.4.4 -> 5.3.1 elasticsearch#24275. Solution: Upgrade to 5.3.3/5.4.1 once they land to get the fix

@LeeDr I haven't tried your latest repro yet, so I don't know which scenario it falls under, or if it's something new.

Once we merge the change to field_caps we should probably close this issue and open up new issues any time a new bug causes this error. This error just means "the field you've selected for this visualization is invalid" and can occur for any number of reasons.

[LeeDr]

@Bargs the issue I hit was caused by a field that is not aggregatable. In our makelogs data there is an extension field and extension.keyword. In a 4.x version of Kibana the Discover tab let you click Visualize on extension but in 5.x you can't because it's not aggregatable (you can on extension.keyword but you have to show hidden fields so you can see that one).

I guess there's at least 2 scenarios here.

1. some saved objects were written directly to .kibana index and when you open that in Kibana you see the error. In this case the bad field is either in the thing you opened or in something it contains. So a dashboard could have multiple saved searches and visualizations which could have a problem. I'm not sure how you would show that info to the user.

2. Importing saved objects can cause the error. In this case maybe we could just improve the error message so that we told the user which field it was and in which saved object.

[Bargs]

Agreed, the error message could be better. And our handling of the scenario where a dashboard has 1 or more invalid saved objects could also be improved. We should probably create a new enhancement request for those tasks, or re-label this issue as an enhancement and create new issues for the legitimate bugs which can lead to this error.

[guyboertje]

@Bargs
I now know the scenario under which I was seeing this. Logstash is getting ready to ship modules like beats. Beats is creating a "full" index_pattern and sending it to ES as part of the module setup and Logstash will too (but did not while I was testing the UI).

Remembering that modules are a "Getting Started Quickly" or a "You are good to go, please don't edit anything (COTS)" solution - an index_pattern refresh when there is insufficient breadth of documents indexed will effectively edit the index_pattern.

In this scenario, the user will only see the error message if:

• the index_pattern constructed by the module author is wrong - ideally we should fix this in the module and the user should upgrade it.
• the unknowing user refreshed the index_pattern with insufficient data - the user should restart the module and have the original index_pattern forcibly restored.
• the knowing user refreshed then edited the index_pattern with insufficient data - the user could restart the module and have the original index_pattern forcibly restored and lose their edits.

This hints at a disconnected index_pattern or with RBAC/Workspaces no edit permissions.

[seliver]

This issue was happening to me because the date field I selected had no data in it (all values where null). Selecting another date field with non-null values solved the problem.

[arch1v1st]

@seliver - thanks for your comment. This was my issue as well!

[fatemabw]

I am also having the exact same issue.

I have E,L,K = 5.5.2

[fatemabw]

This issue has been resolved.

[spanishgum]

I am seeing this issue in ELK 5.6, using the packetbeat import-dashboard script.
Many of the visuals just idle at "loading" indefinitely. Not sure if that is because they do not have a proper field selected or what.

I've tried various tweaks I've found in the forums but can't find a general solution other than fixing a visual's settings one at a time.

An example would be the "Connections over time" visual in "Packetbeat Flows" dash.

It ships a visState like:

{
  "title": "Connections over time",
  "type": "area",
  "params": {
    "shareYAxis": true,
    "addTooltip": true,
    "addLegend": true,
    "legendPosition": "right",
    "smoothLines": true,
    "scale": "linear",
    "interpolate": "linear",
    "mode": "stacked",
    "times": [],
    "addTimeMarker": false,
    "defaultYExtents": false,
    "setYExtents": false,
    "yAxis": {}
  },
  "aggs": [
    {
      "id": "1",
      "enabled": true,
      "type": "cardinality",
      "schema": "metric",
      "params": {
        "field": "flow_id"
      }
    },
    {
      "id": "2",
      "enabled": true,
      "type": "date_histogram",
      "schema": "segment",
      "params": {
        "field": "@timestamp",
        "interval": "auto",
        "customInterval": "2h",
        "min_doc_count": 1,
        "extended_bounds": {}
      }
    }
  ],
  "listeners": {}
}

I went to edit the visual, and selected the metric field to be flow_id.keyword and it works. I'm assuming I would have to do this for every visual to get it all working properly?

[fatemabw]

Hi Adam,

I was able to resolve the issue by editing the JSON code for my dashboard panels, for
which I was getting "field" required error.
My issue was, visualize was looking for a field that was used in building the Viz, but
that field wasn't being indexed , and hence Kibana was not able to find that field in the logs.

I had to do this for every panel in the Viz to fix the problem.

Fatema.

[stacey-gammon]

Running into this very frequently running 6.0.0-rc-1. No imported dashboards or indexes, everything was created on 6.0.0-rc1. Throwing the error constantly on the visualize page, as well as when turning on and off a filter on Discover.

I have quite a few index patterns generated from logstash, packetbeat, metricbeat, etc.

Will hopefully find time to dig in and figure out what the issue is.

[chrisronline]

@spanishgum I'm trying to reproduce this. I started with a fresh 5.6 installation then ran the import dashboards script in packetbeat. After that, I started packetbeat and I'm able to load all visualizations and see the appropriate data. Are there more steps involved?

[bechhansen]

Hi.

I have just updated from Kibana 4.5.4 to 6.0.0 and Elasticsearch 2.3.4 to 6.0.0.
I get the exact same problem when importing 4.5.4 based objects to Kibana 6.0.0.

The data that the imported objects are creating dashboards for is in the logstash-* format.

Is there no way to transfer your dashboards from Kibana 4.5.4 to 6.0.0?

[ghost]

Hi,

What I've noted while playing around with this is that the field name seems to be different. As an example - when importing the dashboards for Winlogbeat it creates one visualization called "Sources". I get the same error when opening the visualization as above. By looking in the Winlogbeat-overview.json file, I see:

"title": "Sources",
        "uiStateJSON": "{}",
        "version": 1,
        "visState": "{\"title\":\"Sources\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"source_name\",\"size\":7,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}"

In the field parameter it's stated source_name, \"field\":\"source_name\". When I edit the visualization and try to find the source_name it's not present, but the source_name.keyword. Choosing that is will make the visualization present what it should. I don't know if this is applicable to all of you, but for me it worked. But it's pretty annoying to update all these visualization manually.

Update
Looking at the index pattern I do have both, source_name and source_name.keyword

This extends my knowledge so maybe this is a stupid question - but why is there two fields for the same data?

Also, I'm running:
Elasticsearch 6.0.0
Logstash 6.0.0
Kibana 6.0.0

[stacey-gammon]

I have not run into this issue recently. Has anyone else on the latest versions of Kibana?

I'm inclined to close this, and wait to see if more bugs reports come in. If so, we can address each one individually, since the original issue was actually fixed.

If anyone feels this is in error, please feel free to comment or re-open.

[gittygoo]

I'm also getting this on the latest versions, pretty similar to what @f-eric said

[apuppy]

I'm also getting this on the latest versions, pretty similar to what @f-eric said

happened again in ELK, version 7.6. Any help?

[LeeDr]

@apuppy Did you only notice it after upgrading? If so, from what version?

[apuppy]

@apuppy Did you only notice it after upgrading? If so, from what version?

The error disappeared when i rerun the following command:

filebeat setup -e

Still don't know why, maybe because of the fields from beats not synchronized to the elasticsearch.

[gittygoo]

I get this alot if i use just "*" as the index pattern to "catch-all" the indexes in order to make some visualizations. It becomes unusable

[0xtf]

Still getting this quite frequently. Seems to be related to a saved search. Running 7.7.0.