Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Ensure annotations index mappings are up to date #61107

Merged

Conversation

droberts195
Copy link
Contributor

When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind. Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

  1. The code that runs on the master node to create the
    annotations index if it doesn't exist but another ML
    index does also now ensures the mappings are up-to-date.
    This is good enough for the ML UI's use of the
    annotations index, because the upgrade order rules say
    that the whole Elasticsearch cluster must be upgraded
    prior to Kibana, so the master node should be on the
    newer version before Kibana tries to write an
    annotation with the new fields.
  2. We now also check whether the annotations index exists
    with the correct mappings before starting an autodetect
    process on a node. This is necessary because ML nodes
    can be upgraded before the master node, so could write
    an annotation with the new fields before the master node
    knows about the new fields.

Relates elastic/kibana#74935

When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Relates elastic/kibana#74935
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@droberts195 droberts195 merged commit a78d951 into elastic:master Aug 13, 2020
@droberts195 droberts195 deleted the ensure_annotations_index_up_to_date branch August 13, 2020 18:05
droberts195 added a commit to droberts195/elasticsearch that referenced this pull request Aug 14, 2020
When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Backport of elastic#61107
droberts195 added a commit to droberts195/elasticsearch that referenced this pull request Aug 14, 2020
When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Backport of elastic#61107
droberts195 added a commit that referenced this pull request Aug 14, 2020
When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Backport of #61107
droberts195 added a commit that referenced this pull request Aug 17, 2020
When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Backport of #61107
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this pull request Aug 24, 2020
This commit removes the log info message "Created ML annotations index and aliases".

The message comes in addition to elasticsearch's index creation logging and it does
not add to it. In addition, since elastic#61107 that message may be logged multiple times.
dimitris-athanasiou added a commit that referenced this pull request Aug 24, 2020
This commit removes the log info message "Created ML annotations index and aliases".

The message comes in addition to elasticsearch's index creation logging and it does
not add to it. In addition, since #61107 that message may be logged multiple times.
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this pull request Aug 24, 2020
…lastic#61461)

This commit removes the log info message "Created ML annotations index and aliases".

The message comes in addition to elasticsearch's index creation logging and it does
not add to it. In addition, since elastic#61107 that message may be logged multiple times.

Backport of elastic#61461
dimitris-athanasiou added a commit that referenced this pull request Aug 24, 2020
…61461) (#61475)

This commit removes the log info message "Created ML annotations index and aliases".

The message comes in addition to elasticsearch's index creation logging and it does
not add to it. In addition, since #61107 that message may be logged multiple times.

Backport of #61461
dimitris-athanasiou added a commit that referenced this pull request Aug 24, 2020
…61461) (#61476)

This commit removes the log info message "Created ML annotations index and aliases".

The message comes in addition to elasticsearch's index creation logging and it does
not add to it. In addition, since #61107 that message may be logged multiple times.

Backport of #61461
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants