[TaskManager] log kibana health report every hour for nodes with background_tasks only role
#158870
Assignees
Labels
enhancement
New value added to drive a business result
Feature:Task Manager
Team:ResponseOps
Label for the ResponseOps team (formerly the Cases and Alerting teams)
Comments
pmuellr
added
enhancement
|
Pinging @elastic/response-ops (Team:ResponseOps) |
github-project-automation
bot
moved this to Awaiting Triage
in AppEx: ResponseOps - Execution & Connectors
pmuellr
moved this from Awaiting Triage
to In Progress
in AppEx: ResponseOps - Execution & Connectors
pmuellr
added a commit
to pmuellr/kibana
that referenced
this issue
Jun 2, 2023
resolves elastic#158870 For Kibana servers that only have node role `background_tasks`, log the task manager health report to the Kibana logs on an interval, currently every hour.
pmuellr
moved this from In Progress
to In Review
in AppEx: ResponseOps - Execution & Connectors
github-project-automation
bot
moved this from In Review
to Done
in AppEx: ResponseOps - Execution & Connectors
pmuellr
added a commit
that referenced
this issue
Jun 6, 2023
…158890) resolves #158870 ## Summary For Kibana servers that only have node role `background_tasks`, log the task manager health report to the Kibana logs on an interval, currently every hour. Co-authored-by: Kibana Machine <[email protected]>
pmuellr
added a commit
to pmuellr/kibana
that referenced
this issue
Jun 6, 2023
…lastic#158890) resolves elastic#158870 ## Summary For Kibana servers that only have node role `background_tasks`, log the task manager health report to the Kibana logs on an interval, currently every hour. Co-authored-by: Kibana Machine <[email protected]> (cherry picked from commit 837ef26) # Conflicts: # x-pack/plugins/task_manager/server/plugin.ts
pmuellr
referenced
this issue
Jun 6, 2023
… role (#158890) (#159142) # Backport This will backport the following commits from `main` to `8.8`: - [[TaskManager] log health on interval with background_tasks only role (#158890)](#158890) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Patrick Mueller","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-06-06T12:42:40Z","message":"[TaskManager] log health on interval with background_tasks only role (#158890)\n\nresolves https://github.com/elastic/kibana/issues/158870\r\n\r\n## Summary\r\n\r\nFor Kibana servers that only have node role `background_tasks`, log the\r\ntask manager health report to the Kibana logs on an interval, currently\r\nevery hour.\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"837ef26fb0cced40214b25f0f1f22a8a0d610fb2","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["enhancement","release_note:skip","Feature:Task Manager","Team:ResponseOps","backport:prev-minor","v8.9.0","v8.8.2"],"number":158890,"url":"https://github.com/elastic/kibana/pull/158890","mergeCommit":{"message":"[TaskManager] log health on interval with background_tasks only role (#158890)\n\nresolves https://github.com/elastic/kibana/issues/158870\r\n\r\n## Summary\r\n\r\nFor Kibana servers that only have node role `background_tasks`, log the\r\ntask manager health report to the Kibana logs on an interval, currently\r\nevery hour.\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"837ef26fb0cced40214b25f0f1f22a8a0d610fb2"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/158890","number":158890,"mergeCommit":{"message":"[TaskManager] log health on interval with background_tasks only role (#158890)\n\nresolves https://github.com/elastic/kibana/issues/158870\r\n\r\n## Summary\r\n\r\nFor Kibana servers that only have node role `background_tasks`, log the\r\ntask manager health report to the Kibana logs on an interval, currently\r\nevery hour.\r\n\r\nCo-authored-by: Kibana Machine <[email protected]>","sha":"837ef26fb0cced40214b25f0f1f22a8a0d610fb2"}},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
One of the issues that has come up since adding support for the Kibana node role
background_tasks, is that when Kibana is deployed with some instances with only that role, and others only withuirole, the Kibana diagnostics bundle generated in the usual fashion willl be missing the runtime data in the task manager health report. That's because the nodes withbackground_tasksare the only nodes actually running tasks, and the nodes withuirole are the only nodes handling http requests, so the diagnostics bundle tool (which uses HTTP APIs to gather data) will only be able to get task manager health for nodes with theuirole.We have a discussion item for long-term solutions, but one simple short-term solution would be to have nodes which only have
background_tasksrunning, generate a health report and log to the Kibana logs. We have some config already which will do this when metrics hit certain thresholds, but this is something we want to do all the time (for now), and not have to have customers configure it - requiring a Kibana restart.The text was updated successfully, but these errors were encountered: