Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exec/Java: Default OOMScoreAdjust causes memory limit to be ignored #10663

Closed
MikeN123 opened this issue May 27, 2021 · 3 comments · Fixed by #10698
Closed

Exec/Java: Default OOMScoreAdjust causes memory limit to be ignored #10663

MikeN123 opened this issue May 27, 2021 · 3 comments · Fixed by #10698
Assignees
@notnoop
Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. type/bug

Comments

@MikeN123
Copy link
Contributor

Nomad version

Nomad v1.1.0 (2678c3604bc9530014208bc167415e167fd440fc)

Operating system and Environment details

Debian Linux 10

Issue

The default setting of OOMScoreAdjust=-1000 in the systemd unit file makes sure Nomad will never be killed by the OOM killer. But, this setting in inherited by child executors and processes, causing the cgroup memory limit not to be enforced. Instead, the OOM killer will log error messages to syslog like crazy, filling up the disk.

Reproduction steps

  1. Use the default systemd unit file.
  2. Use the exec or java driver and start a task that runs over its memory limit.

Expected Result

OOM killer should kick in and kill the process.

I believe the drivers should reset the OOMScoreAdjust for child processes to some sensible value. The only other workaround (at least for the java driver) is to not set an OOMScoreAdjust on the nomad process, but that has the disadvantage that nomad may be killed if the host experiences memory pressure.

Actual Result

the task is not killed, as the OOM killer is unable to kill any process due to the OOMScoreAdjust.

This was also noted in the issue where the default was suggested, but apparently never picked up: #6672 (comment)
The message logged to syslog looks like this:

May 24 21:31:47 d-store-01 kernel: [1775404.602255] Memory limit reached of cgroup /nomad/11bda2ef-ac35-d99a-aaac-d7fb38c21966
May 24 21:31:47 d-store-01 kernel: [1775404.602256] memory: usage 524356kB, limit 524288kB, failcnt 202
May 24 21:31:47 d-store-01 kernel: [1775404.602256] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
May 24 21:31:47 d-store-01 kernel: [1775404.602256] kmem: usage 2980kB, limit 9007199254740988kB, failcnt 0
May 24 21:31:47 d-store-01 kernel: [1775404.602257] Memory cgroup stats for /nomad/11bda2ef-ac35-d99a-aaac-d7fb38c21966: cache:0KB rss:521340KB rss_huge:65536KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:260816KB active_anon:260544KB inactive_file:0KB active_file:0KB unevictable:0KB
May 24 21:31:47 d-store-01 kernel: [1775404.602259] Tasks state (memory values in pages):
May 24 21:31:47 d-store-01 kernel: [1775404.602259] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
May 24 21:31:47 d-store-01 kernel: [1775404.602264] [  20937] 65534 20937   915250   137172  1466368        0         -1000 java
May 24 21:31:47 d-store-01 kernel: [1775404.602264] Out of memory and no killable processes...
May 24 21:31:47 d-store-01 kernel: [1775404.602342] http-nio-8082-e invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-1000
May 24 21:31:47 d-store-01 kernel: [1775404.602343] http-nio-8082-e cpuset=shared mems_allowed=0
May 24 21:31:47 d-store-01 kernel: [1775404.602344] CPU: 0 PID: 20986 Comm: http-nio-8082-e Not tainted 4.19.0-16-amd64 #1 Debian 4.19.181-1
May 24 21:31:47 d-store-01 kernel: [1775404.602344] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
May 24 21:31:47 d-store-01 kernel: [1775404.602344] Call Trace:
May 24 21:31:47 d-store-01 kernel: [1775404.602345]  dump_stack+0x66/0x81
May 24 21:31:47 d-store-01 kernel: [1775404.602346]  dump_header+0x6b/0x283
May 24 21:31:47 d-store-01 kernel: [1775404.602347]  out_of_memory.cold.33+0xa/0x7e
@notnoop
Copy link
Contributor

notnoop commented May 27, 2021

Hi @MikeN123 ! Thanks for raising it. This is indeed a bad behavior and should be fixed. We'll aim to ship a fix soon.

@notnoop notnoop self-assigned this May 27, 2021
@notnoop notnoop added the stage/accepted Confirmed, and intend to work on. No timeline committment though. label May 27, 2021
@MikeN123
Copy link
Contributor Author

Thanks!

I also noted that the PR for memory oversubscription said something on oom_score_adj, not sure if that was ever completed. You may also want to consider varying the oom_score_adj based on the task being oversubscribed or not, K8s does something like that, but maybe that's complicated also keeping in mind Docker and other drivers...

@notnoop notnoop mentioned this issue Jun 3, 2021
Merged
notnoop pushed a commit that referenced this issue Jun 3, 2021
drivers/exec: Don't inherit Nomad oom_score_adj value (#10698)
0be58d7 
Explicitly set the `oom_score_adj` value for `exec` and `java` tasks.

We recommend that the Nomad service to have oom_score_adj of a low value
(e.g. -1000) to avoid having nomad agent OOM Killed if the node is
oversubscriped.

However, Nomad's workloads should not inherit Nomad's process, which is
the default behavior.

Fixes #10663
notnoop pushed a commit that referenced this issue Jun 9, 2021
drivers/exec: Don't inherit Nomad oom_score_adj value (#10698)
9db0909 
Explicitly set the `oom_score_adj` value for `exec` and `java` tasks.

We recommend that the Nomad service to have oom_score_adj of a low value
(e.g. -1000) to avoid having nomad agent OOM Killed if the node is
oversubscriped.

However, Nomad's workloads should not inherit Nomad's process, which is
the default behavior.

Fixes #10663
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@notnoop notnoop

Labels
stage/accepted Confirmed, and intend to work on. No timeline committment though. type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

drivers/exec: Don't inherit Nomad oom_score_adj value hashicorp/nomad
2 participants
@notnoop @MikeN123