Kibana Discovery fails with "closed" when searching in an Index pattern that contains a closed index

[ManofWax]

Kibana version:
7.14.1

Elasticsearch version:
7.14.1

Server OS version:
Ubuntu 20.04

Browser version:
Chrome 93

Browser OS version:
Ubuntu 20.04

Original install method (e.g. download page, yum, from source, etc.):
Official Docker images

Describe the bug:
When I search an index pattern that contains at least one closed index, even if the index does not contains data I'm searching for, the search fails because it looks like that Kibana is trying to do a fields search on every index.

I also read through the Kibana docs to search a config setting that let me do this but I couldn't find anything relevant

Steps to reproduce:

1. Create two or more indexes like foobar-2021-09-09, foobar-2021-09-08 and the Index pattern foobar-*
2. Close foobar-2021-09-08
3. Search in the last 15mins on index pattern foobar-*

Expected behavior:
Kibana should query only the open indices, I'm pretty sure that Kibana 6.x doesn't have this problem.

[ManofWax]

Anyone knows if this is a bug or an intended beheviour? In case this is intended is there a workaround?

[marius-dr]

Hello,
This is sort of an unintended consequence of it.
It didn't happen before 7.11 or 7.12 (can't recall exactly when we introduced using the fields API instead of fields cache).

The workaround for this would be to freeze the index instead of closing it.

Backstory:
we moved from having a field cache saved on the index pattern in order to make the appearance of new fields added to an index faster in Kibana. This was causing a lot of headache for the users. Now the fields are not saved with the index pattern any longer, we just call the field API any time we need it (in Discover we need it for the field bar on the side and a few more other places).

I would see an enhancement request to only warn in case of hitting a closed index instead of direct failure. if you want to add it, be my guest, if you would like me to add it, let me know and I'll do it.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[ManofWax]

My use case is the following: I have a lots of indices for log collection, I keep them for two months before deleting them and storing the data as a snapshot on a backup space

Since my users needs to query only the last day of data 99% of the time, I was closing indices older than 1 week so it didn't waste heap space and resources. I think i could freeze those and gain back some resources.

One thing that's not clear to me about frozen indices is the following: These data structures will need to be reloaded on demand (and subsequently dropped) for each search request that targets the frozen index. That's means that if I query a Kibana index-pattern that contains some froze indices all of them need to be reloaded even if I'm querieng the last 15 minutes of data?

[ManofWax]

Please ignore the last comment, I just read the frozen indices are deprecated on elasticsearch 7.14.
So I guess that I have no alternative and keep them open

[timroes]

I think this is a totally valid use-case and I see why we would want to continue searching over index patterns even ifyou close an index in them. Because at that point they should simply be ignored.

I was unfortunately unable to reproduce the failure as you described it. It seem to actually work fine for me. I tried it with the following commands in the dev tools:

PUT /closed-test-1
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

PUT /closed-test-2
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

POST closed-test-1/_close

I created in between (or also after, tried different combinations) an index pattern for closed-test-*, but I don't see any failures in Discover for it. Neither do I see any failures when trying to query /closed-test-*/_field_caps?fields=* after the index is closed.

@ManofWax @marius-dr can either of you provide me with some more details on how to reproduce it?

[ManofWax]

Ok I understand better the problem, it seems to be related to index aliases: this is how to reproduce:

PUT /closed-test-1
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
    }
  },
  "aliases": {
    "alert": {}
  }
}

PUT /closed-test-2
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
    }
  },
  "aliases": {
    "alert": {}
  }
}

POST closed-test-2/_doc/
{
  "@timestamp": "2021-11-15T13:12:00",
  "message": "GET /search HTTP/1.1 200 1070000",
  "user": {
    "id": "kimchy"
  }
}

POST closed-test-1/_close

Now create a Kibana index-pattern for for alert (that is the index alias we just create). Go to Discover and the following error is shown:

[object Object]: cluster_block_exception: [cluster_block_exception] Reason: index [closed-test-1] blocked by: [FORBIDDEN/4/index closed];

[EDIT]
I did a test in production and I can confirm this only happens when quering index-pattern on index aliases

[timroes]

Thanks for those steps to reproduce. This indeed looks like a bug to me. I've opened an issue in Elasticsearch for this: elastic/elasticsearch#77612

I will close this one here, since to my knowledge there's nothing we could do to workaround that in Kibana.