Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DocDB] Shared memory related core dumps in tserver/pg_gate/postgres processes in aggressive cgroups test #24263

Closed
1 task done
qvad opened this issue Oct 3, 2024 · 0 comments
Assignees
Labels
2024.2 Backport Required area/docdb YugabyteDB core features kind/bug This issue is a bug priority/medium Medium priority issue

Comments

@qvad
Copy link
Contributor

qvad commented Oct 3, 2024

Jira Link: DB-13158

Description

Most likely related to [#23999] DocDB: Big shared memory segments

"sudo mkdir /sys/fs/cgroup/memory/ysql/",
"echo 500444000 | sudo tee /sys/fs/cgroup/memory/ysql/memory.limit_in_bytes",
"sudo chmod 666 /sys/fs/cgroup/memory/ysql/cgroup.procs",
lldb) target create "/home/yugabyte/yb-software/yugabyte-2.25.0.0-b45-almalinux8-aarch64/bin/yb-server" --core "/home/yugabyte/cores/core_45776_1727981414_!home!yugabyte!yb-software!yugabyte-2.25.0.0-b45-almalinux8-aarch64!bin!yb-server"
Core file '/home/yugabyte/cores/core_45776_1727981414_!home!yugabyte!yb-software!yugabyte-2.25.0.0-b45-almalinux8-aarch64!bin!yb-server' (aarch64) was loaded.
(lldb) bt all
* thread #1, name = 'yb-tserver', stop reason = signal SIGSEGV: address not mapped to object
  * frame #0: 0x0000aaaab4fb4820 yb-server`yb::tserver::PgClientSession::ScheduleBigSharedMemExpirationCheck(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>) [inlined] std::__1::unique_ptr<yb::rpc::Scheduler::Impl, std::__1::default_delete<yb::rpc::Scheduler::Impl>>::operator->[abi:ue170006](this=0x0000000000000000) const at unique_ptr.h:277:19
    frame #1: 0x0000aaaab4fb4820 yb-server`yb::tserver::PgClientSession::ScheduleBigSharedMemExpirationCheck(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>) [inlined] yb::rpc::Scheduler::Abort(this=0x0000000000000000, task_id=0) at scheduler.cc:182:3
    frame #2: 0x0000aaaab4fb481c yb-server`yb::tserver::PgClientSession::ScheduleBigSharedMemExpirationCheck(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>) [inlined] yb::rpc::ScheduledTaskTracker::Abort(this=<unavailable>) at scheduler.cc:203:17
    frame #3: 0x0000aaaab4fb480c yb-server`yb::tserver::PgClientSession::ScheduleBigSharedMemExpirationCheck(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>) [inlined] void ...

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@qvad qvad added area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Oct 3, 2024
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Oct 3, 2024
@rthallamko3 rthallamko3 removed the status/awaiting-triage Issue awaiting triage label Oct 4, 2024
spolitov added a commit that referenced this issue Oct 7, 2024
Summary:
It could happen that pg client session is destroyed while `big_shared_mem_expiration_task_` is scheduled to run.
As result it could be executed on destroyed object.
Jira: DB-13158

Test Plan: Jenkins

Reviewers: rthallam, esheng

Reviewed By: esheng

Subscribers: ybase, yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D38688
spolitov added a commit that referenced this issue Oct 9, 2024
… before destroying it

Summary:
It could happen that pg client session is destroyed while `big_shared_mem_expiration_task_` is scheduled to run.
As result it could be executed on destroyed object.
Jira: DB-13158
Original commit: 1e0cdcd/D38688

Test Plan: Jenkins

Reviewers: rthallam, esheng

Reviewed By: esheng

Subscribers: yql, ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D38795
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2024.2 Backport Required area/docdb YugabyteDB core features kind/bug This issue is a bug priority/medium Medium priority issue
Projects
None yet
Development

No branches or pull requests

4 participants