Dotted named field values can merge with non-dotted fields of the same name

[n0othing]

Elasticsearch version (bin/elasticsearch --version): version: 7.10.0, Build: default/tar/51e9d6f22758d0374a0f3f5c6e8f3a7997850f96/2020-11-09T21:30:33.964949Z, JVM: 15.0.1

Plugins installed: []

JVM version (java -version): JVM: 15.0.1 (bundled)

OS version (uname -a if on a Unix-like system): macOS 10.15.7

Description of the problem including expected versus actual behavior:

When using dynamic: false a field with a dot in its name, whose suffix matches a mapped field’s name has its values merged with the mapped field unexpectedly:

Steps to reproduce:

DELETE myindex

PUT myindex
{
  "mappings": {
    "dynamic": false,
    "properties": {
      "myfield": {
        "type": "keyword"
      }
    }
  }
}

PUT myindex/_doc/1?refresh=true
{
  "myfield": "foo",
  "something.myfield": [
    "bar",
    "baz"
  ]
}


GET myindex/_search?size=0
{
  "aggs": {
    "my_term_agg": {
      "terms": {
        "field": "myfield",
        "size": 10
      }
    }
  }
}

Results in:

# GET myindex/_search?size=0
{
  "took" : 22,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "my_term_agg" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "bar",
          "doc_count" : 1
        },
        {
          "key" : "baz",
          "doc_count" : 1
        },
        {
          "key" : "foo",
          "doc_count" : 1
        }
      ]
    }
  }
}

In the above, something.myfield's values are getting added to myfield. This doesn't happen if something.myfield is mapped/indexed.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jtibshirani]

Thanks @n0othing for raising this, it's a great catch. In addition to the requirements you listed, it seems this only happens if the dynamic field value is an array.

I think this bug has existed for a long time, I tested on 6.8 and saw the same issue. Here's the piece of code that seems to be off: DocumentParser.java#L573.

[javanna]

This looks to have been fixed by #79922 . Opened #85081 to add a test around it.