ES Indices /GET api return closed indices if they have aliases assigned.

[SthPhoenix]

Elasticsearch version: 7.1.0, 7.2.0

Plugins installed: [analysis-icu, analysis-smartc, analysis-phonetic]

JVM version : bundled

OS version: Ubuntu 18.04.2

Description of the problem including expected versus actual behavior:

Actual: Elasticsearch Indices /GET API return closed indices if they has assigned aliases.
Expected: ES should return only opened indices.

Aliases are unique to closed indices, and there are no opened indices sharing same aliases.

Steps to reproduce:

1. Create new index
2. Put alias to this index
3. Close index
4. Execute GET http://elasticsearch.host:9200/* to get list of indices including closed just now.
5. Delete alias from closed index.
6. Execute GET http://elasticsearch.host:9200/* to get list of indices without closed one.

[elasticmachine]

Pinging @elastic/es-distributed

[dliappis]

Hey @SthPhoenix ,

Thanks for this bug report.

I am not sure I see the problem here. The most typical API to get a list of indices is GET _cat/indices (or GET _aliases). Using either API the state of the index after closing it seems correct to me whether there is an alias or not.

Testing closing an index without an alias shows:

DELETE testindex

PUT testindex

GET /_cat/indices
yellow open testindex            ygpMPnV1R1SCc-DwP6-IlQ 1 1 0 0   230b   230b

GET /_aliases
{
  "testindex" : {
    "aliases" : { }
  }
}

POST /testindex/_close

GET /_cat/indices
yellow close testindex            ygpMPnV1R1SCc-DwP6-IlQ 1 1                  

GET /_aliases
{
  "testindex" : {
    "aliases" : { }
  }
}

Testing closing an index with an alias shows:

DELETE testindex

PUT testindex

POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "testindex", "alias": "testalias"
      }
    }]
}

GET /_cat/indices
yellow open testindex            fAq8AxBASpasUPJdAE2OmA 1 1 0 0   230b   230b

GET /_aliases
{
"testindex" : {
    "aliases" : {
      "testalias" : { }
    }
  }
}

POST /testindex/_close

GET /_cat/indices
yellow close testindex            fAq8AxBASpasUPJdAE2OmA 1 1                  

GET /_aliases
{
  "testindex" : {
    "aliases" : {
      "testalias" : { }
    }
  }
}

[SthPhoenix]

Yes, _cat API works as intended. Problem is that Get Index API have inconsistent behavior.

Actually I noticed it while using official python client, where es.indicies.get('*') in previous versions of ES returned only opened indices, and after 7.1 it start returning closed ones.

[dliappis]

@SthPhoenix I can't reproduce this (on 7.2.0 at least):

I tried:

1. Created index (PUT testindex)
2. Assign alias

   POST /_aliases
   {
     "actions": [
       {
         "add": {
           "index": "testindex", "alias": "testalias"
         }
       }]
   }
   

3. Check GET _all (or GET *) and shows the open index as expected.
4. Close index (POST /testindex/_close)
5. Check GET _all (or GET *) and the closed index testindex isn't listed.

Are these the same reproduction steps you followed?

[SthPhoenix]

I am using python client (7.0.0) for this, but yes, steps are the same. For checking indices list I also used curl.
Tested today on ES 7.3, got the same bug.
If it might be the case, this is not clean install, ES was updated step by step from ES 6.5.1

[dliappis]

For checking indices list I also used curl.

To clarify: if you follow my earlier steps using curl, does step 5. i.e. curl -H 'Content-Type:application/json' <es_url:es_port>/_all?pretty list testindex?

[SthPhoenix]

Yes, using curl gives same strange behavior.
Also tested this with clean docker containers 7.1 to 7.3 - and it works just as expected, that's weird. I'll test it more thoroughly and let you know if I could figure out the reason.

[tlrx]

@SthPhoenix Did you manage to reproduce or to figure out the reason?

Looking at the code, the GET Indices API should only return opened indices on 6.x, 7.x and master as it uses the "strict expand open" option.

It is possible that the Elasticsearch Python Client sets a different indices options that allow closed indices to be included in the response, but looking at the client code it does not seem to be the case.

Also, I'm curious to know if any of the clusters you tested have the Security enabled?

[SthPhoenix]

Hi @tlrx ! I won't have access to the cluster with this error till the end of next week.

It is possible that the Elasticsearch Python Client sets a different indices options that allow closed indices to be included in the response, but looking at the client code it does not seem to be the case.

Since this behavior is also reproduced with curl requests, I think ES python client is out of question.

Also, I'm curious to know if any of the clusters you tested have the Security enabled?

Yes, I have security enabled on the cluster.
I have updated to ES 7.1, enabled security and later found this issue, but disabling security didn't help.

[tlrx]

@SthPhoenix Thanks for the extra information.

Since this behavior is also reproduced with curl requests, I think ES python client is out of question.

Agreed, I just wanted to double check.

Yes, I have security enabled on the cluster.

That's what I suspected, and indeed this bug can be reproduced on 7.1/7.2 (maybe also later versions) but it only appears when security is enabled.

[elasticmachine]

Pinging @elastic/es-security

[SthPhoenix]

I have checked it on a single node with security. Yes, it looks like the bug can be reproduced only with security enabled.
Also GET API returns closed indices only if mask applies to alias, i.e. if index name is test_index and it's alias is test_alias index will be returned for masks like: *, test*, test_*, and won't be returned for masks like test_i*

[SthPhoenix]

Connected bug found:
Search requests to _all indices or with mask which fit to alias of closed index fails with "index closed exception"

[tvernum]

This is the same root problem as #32238.

Unfortunately this is a limitation when security is enabled which has been around, essentially forever and is very hard to fix unless we either:

1. change the security model to not respect privileges on aliases
2. redesign wildcard resolution in core ES to be pluggable (so security can do just-in-time expansion, which might make some of these cases a bit simpler).

[albertzaharovits]

Another option is to change how expand_wildcards works in core (breaking change) so as to make it compatible with how Security currently functions.

Wildcards that match aliases that point to both closed and open indices can be made to expand to only open, closed indices or both . To honor this option ES core needs wildcards in the index expression . But the Security Filter cannot forward wildcards.

We could deprecate expand_wildcards in favor of expand_aliases with a similar behavior but for aliases not wildcards.