Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Ensure data frame analytics jobs don't run on a node that's too new #61325

Closed
droberts195 opened this issue Aug 19, 2020 · 2 comments
Closed
Labels
:ml Machine learning

Comments

@droberts195
Copy link
Contributor

PR #43880 "adds the analytics version in the persistent task params as this allows us to prevent tasks to run on unsuitable nodes in the future."

Due to #61104 the future is now.

Nodes running 7.10 or above must not be allowed to be assigned data frame analytics jobs whose persistent task params were created in 7.9 or earlier, as the mappings on the destination index will be for the pre-#61104 results format.

Nodes running 7.9 or earlier must not be allowed to be assigned data frame analytics whose persistent task params were created in 7.10 or later, as the mappings on the destination index will be for the post-#61104 results format.

Those two rules nicely cover jobs that are running during rolling upgrade and need to be moved to a different node part way through. There is another case where having the version in the persistent task params doesn't help, and that is if someone attempts to start a job in 7.10 or above that was manually stopped part way through its run in 7.9 or earlier. We need to prevent this too. We store the version that created the destination index in its metadata. Hopefully we can use this to prevent a job being started if its destination index is too old for the latest results format.

@droberts195 droberts195 added the :ml Machine learning label Aug 19, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@przemekwitek
Copy link
Contributor

PRs #62749 and #62960 together solve this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:ml Machine learning
Projects
None yet
Development

No branches or pull requests

3 participants