With ES 7.10.2 unable to make geo_point field not-indexed (Regression from 7.5.2)

[plebedev]

Elasticsearch version (bin/elasticsearch --version):
Version: 7.10.2, Build: default/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z, JVM: 11.0.6

Plugins installed: []

JVM version (java -version):
java version "11.0.6" 2020-01-14 LTS

OS version (uname -a if on a Unix-like system):
18.7.0 Darwin Kernel Version 18.7.0: Fri Oct 30 12:37:06 PDT 2020; root:xnu-4903.278.44.0.2~1/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:
When creating a mapping with a geo_point field and "index":false, the index property gets ignored. The expected behavior is that that field is not getting index, and "index" property is retained in the mapping. The other problem is adding any new field to the mapping fails.

Steps to reproduce:

1. Create an index:
   curl -XPUT "localhost:9200/test_index"
2. Put a mapping:

curl -XPUT "localhost:9200/test_index/_mapping" --header "Content-Type: application/json" -d '
{
 "properties": {
   "email": {
     "type": "keyword",
     "index" : false
   },
   "location" : {
     "type" : "geo_point",
     "index" : false
    }
 }
}'

3. Check the mapping:

curl -X GET "localhost:9200/test_index/_mapping?pretty"
{
  "test_index" : {
    "mappings" : {
      "properties" : {
        "email" : {
          "type" : "keyword",
          "index" : false
        },
        "location" : {
          "type" : "geo_point"
        }
      }
    }
  }
}

As you can see, "index":false is missing for location field.
4. now add a new field to the mapping:

curl -XPUT "localhost:9200/test_index/_mapping?pretty" --header "Content-Type: application/json" -d '
{
  "properties": {
    "email": {
      "type": "keyword",
      "index" : false
    },
    "location" : {
      "type" : "geo_point",
      "index" : false
     },
     "age": {
      "type": "integer",
      "index" : false
    }
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Mapper for [location] conflicts with existing mapping:\n[mapper [location] has different [index] values, mapper [location] has different [index_options] values]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Mapper for [location] conflicts with existing mapping:\n[mapper [location] has different [index] values, mapper [location] has different [index_options] values]"
  },
  "status" : 400
}

As you can see, even though the only difference between two mapping requests is the addition of a new field, the request fails because the created mapping in (2) is different from what was actually in the mapping request.

The same steps work in ES 7.5.2, the output of step 3 there is

curl -X GET "localhost:9200/test_index/_mapping?pretty"
{
  "test_index" : {
    "mappings" : {
      "properties" : {
        "email" : {
          "type" : "keyword",
          "index" : false
        },
        "location" : {
          "type" : "geo_point",
          "index" : false
        }
      }
    }
  }
}

and step 4 also works:

curl -XPUT "localhost:9200/test_index/_mapping?pretty" --header "Content-Type: application/json" -d '
{
"properties": {
  "email": {
    "type": "keyword",
    "index" : false
  },
  "location" : {
    "type" : "geo_point",
    "index" : false
  },
  "age": {
    "type": "integer",
     "index" : false
   }
 }
}'
{
"acknowledged" : true
}

curl -X GET "localhost:9200/test_index/_mapping?pretty"
{
"test_index" : {
  "mappings" : {
    "properties" : {
      "age" : {
        "type" : "integer",
        "index" : false
      },
      "email" : {
        "type" : "keyword",
        "index" : false
      },
      "location" : {
        "type" : "geo_point",
        "index" : false
      }
    }
  }
 }
}

[plebedev]

I also tested it with 7.8.1 and it worked but did not work with 7.9.0, so looks like a regression in 7.9.0.

My gut feeling is that this change broke it but I have no proof: #58160
There are a changes for a lot of field types, including AbstractGeometryFieldType and the AbstractGeometryFieldMapper .Builder() has protected boolean indexed = true; and it does not look like that there is a way to change it

[iverase]

Hi @plebedev,

Yes, you are right there is a regression in 7.9.x and 7.10.x but it seems to have been fixed in 7.11.x. I suspect due to the following change: #63836. You will need to upgrade to 7.11 to address the regression.

There is a hole in the testing that needs to be addressed.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[plebedev]

Indeed, this is working in 7.11. However, there was a significant license change in 7.11, and it is no no longer a straight forward upgrade from 7.5.2 for us. We need to get it clear with a legal department :(

Is there a change the fix can be back-ported, and 7.10.3 is released?

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[plebedev]

We also have a custom ES plug-in that extends painless. We use lang-painless, that seems to disappear in 7.11 and this is not mentioned as a breaking change. This makes migration to 7.11 much more complicated. We can change it to use elasticsearch-scripting-painless-spi instead but now we get errors for grants that we did not have before.

This is not related to this ticket directly but given that there is no clear documentation of what changed wrt plug-in development in 7.11, it complicates the migration.

[iverase]

Sorry for the late reply but we discuss it internally. It is not possible to release a 7.10.x version anymore due to backward compatibilities constraints. I am afraid the way forward is to upgrade to 7.11+.