[ML] After 7.3->7.4 upgrade get transforms only shows transforms from 7.3

[droberts195]

(Found by @wwang500.)

1. Install 7.3
2. Create some data frame transforms
3. Upgrade to 7.4
4. Create some more data frame transforms
5. GET _data_frame/transforms
6. GET _data_frame/transforms/_stats

The problem is that steps 5 and 6 only show the data frame transforms created in step 2. The data frame transforms in step 4 are invisible to these endpoints.

The root cause of the problem can be seen by running these searches:

GET .data-frame-internal-*/_search?q=doc_type:data_frame_transform_config

GET .data-frame-internal-*/_search?q=doc_type:data_frame_transform_config&sort=id

In the case of @wwang500's Cloud cluster the first search returns 10 documents and the second search returns 3 documents and this warning:

  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 1,
    "failures" : [
      {
        "shard" : 0,
        "index" : ".data-frame-internal-2",
        "node" : "azom_s-7RzC_59GZv4cm8g",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
        }
      }
    ]
  },

Calling GET .data-frame-internal-2/_mapping shows that the problem is that a document was written to the .data-frame-internal-2 index before the corresponding index template was created, so it has been created with dynamic mappings.

It means there must be some loophole where the index template that is supposed to be installed as soon as a 7.4 node starts up does not get installed immediately, and there is enough time for a transform that was running during the upgrade to get assigned to the 7.4 node and write a document to the .data-frame-internal-2 index before the template is installed.

GET .data-frame-internal-2/_search?seq_no_primary_term=true&q=_seq_no:0 shows that the first document written to the .data-frame-internal-2 index was a checkpoint document:

    ...
    "hits" : [
      {
        "_index" : ".data-frame-internal-2",
        "_type" : "_doc",
        "_id" : "data_frame_transform_checkpoint-dfm-continuous-2",
        "_seq_no" : 0,
        "_primary_term" : 1,
        "_score" : 1.0,
        "_source" : {
          "id" : "dfm-continuous",
          "checkpoint" : 2,
          "doc_type" : "data_frame_transform_checkpoint",
          ...

So it's the place where checkpoints are written that can sneak ahead of the index template upgrade.

[elasticmachine]

Pinging @elastic/ml-core