Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix worker OOM #9949

Merged
merged 1 commit into from
Jan 11, 2022
Merged

Fix worker OOM #9949

merged 1 commit into from
Jan 11, 2022

Conversation

XuPengfei-1020
Copy link
Member

The io.trino.orc.reader.SliceDictionaryColumnReader#dictionaryData could retaine hundreds MB memory.
Sometime caused worker OOM.

@cla-bot cla-bot bot added the cla-signed label Nov 12, 2021
@sopel39 sopel39 requested a review from skrzypo987 November 13, 2021 21:04
@XuPengfei-1020 XuPengfei-1020 force-pushed the master branch 2 times, most recently from ccd462b to 2f150a9 Compare December 23, 2021 12:51
@sopel39 sopel39 requested a review from losipiuk December 23, 2021 14:59
@cla-bot
Copy link

cla-bot bot commented Dec 23, 2021

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: [email protected].
This is most likely caused by a git client misconfiguration; please make sure to:

  1. check if your git client is configured with an email to sign commits git config --list | grep email
  2. If not, set it up using git config --global user.email [email protected]
  3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

@cla-bot cla-bot bot removed the cla-signed label Dec 23, 2021
@cla-bot cla-bot bot added the cla-signed label Dec 23, 2021
@XuPengfei-1020 XuPengfei-1020 force-pushed the master branch 2 times, most recently from cfd03d9 to d992637 Compare January 5, 2022 13:04
@XuPengfei-1020 XuPengfei-1020 requested a review from sopel39 January 5, 2022 15:02
sopel39
sopel39 approved these changes Jan 5, 2022
View reviewed changes
Copy link
Member

@sopel39 sopel39 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm % comments

lib/trino-orc/src/main/java/io/trino/orc/reader/SliceDictionaryColumnReader.java Show resolved Hide resolved
lib/trino-orc/src/main/java/io/trino/orc/reader/SliceDictionaryColumnReader.java Show resolved Hide resolved
lib/trino-orc/src/test/java/io/trino/orc/TestSliceDictionaryColumnReader.java Outdated Show resolved Hide resolved
lib/trino-orc/src/test/java/io/trino/orc/TestSliceDictionaryColumnReader.java Outdated Show resolved Hide resolved
@XuPengfei-1020
Account for memory usage of dictionary in SliceDictionaryColumnReader
b8aec8b 
SliceDictionaryColumnReader#dictionaryData could retain hundreds
MB memory, sometime can cause worker OOM, especially in the
situation of small worker and large concurrency.
@sopel39 sopel39 merged commit f722686 into trinodb:master Jan 11, 2022
@sopel39 sopel39 mentioned this pull request Jan 11, 2022
Closed
@github-actions github-actions bot added this to the 368 milestone Jan 11, 2022
@mosabua mosabua mentioned this pull request Jan 11, 2022
Merged
@maltesander maltesander mentioned this pull request Apr 12, 2022
Closed
7 tasks
@MnO2 MnO2 mentioned this pull request Mar 8, 2023
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sopel39 sopel39 sopel39 approved these changes

@dain dain Awaiting requested review from dain

@skrzypo987 skrzypo987 Awaiting requested review from skrzypo987

@losipiuk losipiuk Awaiting requested review from losipiuk

@martint martint Awaiting requested review from martint

@raunaqmorarka raunaqmorarka Awaiting requested review from raunaqmorarka

Assignees
No one assigned
Labels
cla-signed
Milestone
368
Development

Successfully merging this pull request may close these issues.
5 participants
@XuPengfei-1020 @martint @raunaqmorarka @sopel39 @skrzypo987