[ILM] ilm-enabled indices use spamming log with messages "applying create index request using v1 templates"

[pmuellr]

About a week ago, the elasticsearch build we are using for Kibana dev started logging a lot of duplicate messages for indices that have ILM policies associated with them. At least I believe that's the common thread for the relevant messages / indices.

Here's the start of the elasticsearch log to show the version info:

elasticsearch log

$ yarn es snapshot --ssl --license trial
yarn run v1.22.0
$ node scripts/es snapshot --ssl --license trial
 info Installing from snapshot
   │ info version: 8.0.0
   │ info install path: /Users/pmuellr/Projects/elastic/kibana/.es/8.0.0
   │ info license: trial
   │ info Downloading snapshot manifest from https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.0.0/manifest-latest-verified.json
   │ info verifying cache of https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.0.0/archives/20200414-194134_0e55fdeae97/elasticsearch-8.0.0-SNAPSHOT-darwin-x86_64.tar.gz
   │ info cache invalid, redownloading
   │ info downloading artifact checksum from https://storage.googleapis.com/kibana-ci-es-snapshots-daily/8.0.0/archives/20200414-194134_0e55fdeae97/elasticsearch-8.0.0-SNAPSHOT-darwin-x86_64.tar.gz.sha512
   │ info checksum verified
   │ info install directory already exists, removing
   │ info extracting /Users/pmuellr/Projects/elastic/kibana/.es/cache/elasticsearch-8.0.0-SNAPSHOT-darwin-x86_64.tar.gz
   │ info extracted to /Users/pmuellr/Projects/elastic/kibana/.es/8.0.0
   │ info created /Users/pmuellr/Projects/elastic/kibana/.es/8.0.0/ES_TMPDIR
   │ info setting secure setting bootstrap.password to changeme
 info Starting
   │ info moved /Users/pmuellr/Projects/elastic/kibana/packages/kbn-dev-utils/certs/elasticsearch.p12 in config to /Users/pmuellr/Projects/elastic/kibana/.es/8.0.0/config/elasticsearch.p12
   │ debg bin/elasticsearch -E xpack.security.http.ssl.enabled=true -E xpack.security.http.ssl.keystore.path=elasticsearch.p12 -E xpack.security.http.ssl.keystore.type=PKCS12 -E xpack.security.http.ssl.keystore.password=storepass
   │ info [o.e.n.Node] [pmuellr.muellerware.org] version[8.0.0-SNAPSHOT], pid[90178], build[default/tar/0e55fdeae97e1e2575b4d1c27574aac82ae4423f/2020-04-14T19:33:51.881372Z], OS[Mac OS X/10.14.6/x86_64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14/14+36]
   │ info [o.e.n.Node] [pmuellr.muellerware.org] JVM home [/Users/pmuellr/Projects/elastic/kibana/.es/8.0.0/jdk.app/Contents/Home]
   │ info [o.e.n.Node] [pmuellr.muellerware.org] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/Users/pmuellr/Projects/elastic/kibana/.es/8.0.0/ES_TMPDIR, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/pmuellr/Projects/elastic/kibana/.es/8.0.0, -Des.path.conf=/Users/pmuellr/Projects/elastic/kibana/.es/8.0.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
   │ info [o.e.n.Node] [pmuellr.muellerware.org] version [8.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
...

After that, every 10 minutes the following line is logged (note, I've expanded the JSON included at the end of the message, and removed the mappings property from it) :

│ info [o.e.c.m.MetadataCreateIndexService] [pmuellr.muellerware.org] applying create index request using v1 templates 

[
  {
    ".kibana-event-log-8.0.0-template": {
      "order": 0,
      "index_patterns": [
        ".kibana-event-log-8.0.0-*"
      ],
      "settings": {
        "index": {
          "lifecycle": {
            "name": ".kibana-event-log-policy",
            "rollover_alias": ".kibana-event-log-8.0.0"
          },
          "number_of_shards": "1",
          "number_of_replicas": "1"
        }
      },
      "mappings": {
      	...
      },
      "aliases": {}
    }
  }
]  

Currently, two log lines are being printed every 10 minutes, one for ilm-history and another for .kibana-event-log-8.0.0.

This was reported by other Kibana users, here: elastic/kibana#63365

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[pmuellr]

It appears the message being logged was added in PR #54669

[dakrone]

Yeah this was me, I have been meaning to adjust the log level to be lower for this, sorry this is causing consternation, I will fix it.

[dakrone]

I fixed this as part of #55411, so I'm closing this issue.