Expand template "index_pattern" matching to allow negative matching

[simozanin]

Following from this discussion: https://discuss.elastic.co/t/template-index-pattern-matching/249289
And this: https://discuss.elastic.co/t/defining-index-mappings-on-elasticsearch/248774/6

It would be helpful to add a way to define templates on all new indices in ES, except those matching a condition. Right now the only way to do this is to use wildcard matching, which is quite difficult to apply in real-case scenarios.

Let's say I want to store documents that represent movies in my ElasticSearch engine, and I want to store each movie in a different index for each production company. I would have indices for warnerbros, disney, 20thcentury, etc. I know I won't store any other kind of object in my engine, so I want all indices to share the same mapping. So it would make sense to define a template right after spinning up the ElasticSearch instance, so that any new index created when indexing a document has the mappings defined in my template.
Right now I have two ways of doing this:

• either I prepend a common string to all the index names, so that I store them in, say: m-warnerbros, m-disney, m-20thcentury, and then define the template with the field: "index_patterns": ["m-*"]
• or I list all the production companies in the template request, like "index_patterns": ["warnerbros", "disney", "..."]

The first solution is not very practical, it forces me to manipulate the index name, and if I want to infer the index name automatically from the contents of my documents this won't work. The other solution is tedious and won't scale well if I need to add new productions companies that didn't previously exist.

Using a pattern matching with just the wildcard, i.e. "index_patterns": ["*"], causes problems with the .log* and .metrics* indices, and if I define an alias in the template then it applies to any new index created, including logs and metrics.

My suggestion is to allow negative matching, so that I can define a template on all new indices that do not match a certain pattern. This way I could exclude the logs and metrics indices from the template. Something along the lines of:

"index_patterns_match": ["*"],
"index_patterns_exclude": [".*", "-*",]

[mayya-sharipova]

@simozanin Is your intention only to exclude what we call hidden indices (usually indices that start with .)? If yes, this feature is already implemented from 7.7, where you can mark you index as hidden, and it will be excluded from a wildcard extension operation, one of which is global index template.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[simozanin]

Hi,

Yes, that is very helpful actually, and should solve my problem. However, I have encountered an issue before where my template was applied to some of the hidden indices and prevented Kibana from starting-up properly, please see log attached.
kibana.log

I have again set the template with the usual PUT request, this is the output of GET _template/product_template:

{
  "product_template" : {
    "order" : 0,
    "index_patterns" : [
      "*"
    ],
    "settings" : {
      "index" : {
        "analysis" : {
          "filter" : {
            "autocomplete_filter" : {
              "type" : "edge_ngram",
              "min_gram" : "1",
              "max_gram" : "20"
            }
          },
          "analyzer" : {
            "autocomplete" : {
              "filter" : [
                "lowercase",
                "autocomplete_filter"
              ],
              "type" : "custom",
              "tokenizer" : "standard"
            }
          }
        }
      }
    },
    "mappings" : {
      "dynamic" : false,
      "properties" : {
        "Ticker" : {
          "type" : "keyword"
        },
        "suggest" : {
          "search_analyzer" : "standard",
          "analyzer" : "autocomplete",
          "type" : "completion"
        },
        "ISIN" : {
          "type" : "keyword"
        },
        "Name" : {
          "type" : "keyword"
        },
        "Cusip" : {
          "type" : "keyword"
        }
      }
    },
    "aliases" : {
      "product" : { }
    }
  }
}

As you can see below, this template is applied to the index .kibana as well:
GET /.kibana

{
  ".kibana" : {
    "aliases" : {
      "product" : { }
    },
    "mappings" : {
      "dynamic" : "false",
      "properties" : {
        "Cusip" : {
          "type" : "keyword"
        },
        "ISIN" : {
          "type" : "keyword"
        },
        "Name" : {
          "type" : "keyword"
        },
        "Ticker" : {
          "type" : "keyword"
        },
        "suggest" : {
          "type" : "completion",
          "analyzer" : "autocomplete",
          "search_analyzer" : "standard",
          "preserve_separators" : true,
          "preserve_position_increments" : true,
          "max_input_length" : 50
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "provided_name" : ".kibana",
        "creation_date" : "1600730912696",
        "analysis" : {
          "filter" : {
            "autocomplete_filter" : {
              "type" : "edge_ngram",
              "min_gram" : "1",
              "max_gram" : "20"
            }
          },
          "analyzer" : {
            "autocomplete" : {
              "filter" : [
                "lowercase",
                "autocomplete_filter"
              ],
              "type" : "custom",
              "tokenizer" : "standard"
            }
          }
        },
        "number_of_replicas" : "1",
        "uuid" : "UrouXhICTYyAlv_i0gde5g",
        "version" : {
          "created" : "7080199"
        }
      }
    }
  }
}

Is it worth opening an bug fix request for this? Or is this expected behaviour?
I am running ElasticSearch and Kibana 7.8.1 on Docker

[lucabelluccini]

While the feature about exclusion/inclusion of patterns might be interesting, I want to comment about the specific issue you're facing with Kibana.
The .kibana index (and other internal system) will become one of the system indices (see #50251) in the future.
Hidden indices (#50452) do not obey to global match all templates (*) but they follow index templates with matching patterns (e.g. .watcher-history*). They were introduced in 7.7.0 as Mayya pointed out.

At the moment, the system indices are not implemented, so Kibana can still be affected by index templates.
In the specific case of Kibana, we're improving the documentation on Kibana (elastic/kibana#80283) to explain how we can avoid the problem you've mentioned above.

[OranShuster]

Looking at the kibana upgrade docs it says to "narrow down index patterns"
We have a very basic use case that does not allow us to do that

We have many indices which contain different type of documents but themselves are very small (<1 gig per day)
We want to have a base ILM for all these indices to we created a base-ilm with some sane defaults and we need to hook it up to the template. the only option is to have a base template (with an added @timestamp date field mapping) so every index that does not have an explicit template + ILM related to it will use the default one automatically

I also can't mark those indices as hidden since the monitoring indices are created each day
The only solution i see here is to create a template with higher priority to match .monitoring indices so they won't be affected
Edit - doing that will make it so the legacy index templates won't be applied and will cause errors in kibana (since the mapping is incorrect)

Is there no way to create a default ILM?