Annotated highlight not maching when search contains both annotation and annotated term

[thelink2012]

Elasticsearch Version

v8.5.2

Installed Plugins

mapper-annotated-text

Java Version

bundled

OS Version

Linux 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

Hi,

We use the mapper-annotated-text plugin in production to tag named-entities and found an particularly bad behavior during highlighting.

Basically, if we have a text aaaa [bbbb cccc](annotated) dddd, and search for bbbb annotated we'll get the highlight aaaa [bbbb](_hit_term=bbbb) cccc dddd where we'd expect bbbb cccc due to annotated being in the query too.

On the other hand, searching for cccc annotated produces the expected result of aaaa [bbbb cccc](_hit_term=annotated&annotated) dddd.

This happens because the annotation process receives the passage matches sorted by ascending offset order, and when a match overlaps with a previously seen one, it ignores it. So in the first case, it sees bbbb first and when it sees annotated it discards it because it overlaps with bbbb. On the latter case, cccc is the one discarded since it appears later in the list of offsets.

We've prototype a fix in this branch and would be glad to iterate on it and advance with a PR if this is considered as an issue.

Steps to Reproduce

PUT example
{
  "mappings": {
    "properties": {
      "body": {
        "type": "annotated_text"
      }
    }
  }
}
POST example/_doc
{
  "body": "aaaa [bbbb cccc](annotated) dddd"
}
POST example/_search
{
  "query": {
    "match_all": {}
  },
  "highlight": {
    "type": "annotated",
    "order": "score",
    "fields": {
      "body": {
        "highlight_query": {
          "bool": {
            "must": {
              "match": {
                "body": {
                  "query": "bbbb annotated"
                }
              }
            }
          }
        }
      }
    }
  }
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)