Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lightning oom when importing wide table on default config #43853

Closed
Tracked by #43740
D3Hunter opened this issue May 16, 2023 · 0 comments · Fixed by #43870
Closed
Tracked by #43740

lightning oom when importing wide table on default config #43853

D3Hunter opened this issue May 16, 2023 · 0 comments · Fixed by #43870
Labels
component/lightning This issue is related to Lightning of TiDB. severity/moderate type/feature-request Categorizes issue or PR as related to a new feature.

Comments

@D3Hunter
Copy link
Contributor

D3Hunter commented May 16, 2023 •
edited
Loading

if the table is wide, say 100k per kv, when writing to tikv using default config, lightning might oom.

current workaround is to use a smaller send-kv-pairs, but we should add a limit in bytes.

img_v2_0036b1dc-e2f2-4643-ba23-68609800cb4g
@D3Hunter D3Hunter mentioned this issue May 16, 2023
Open
16 tasks
@D3Hunter D3Hunter added component/lightning This issue is related to Lightning of TiDB. type/bug The issue is confirmed as a bug. severity/moderate type/feature-request Categorizes issue or PR as related to a new feature. and removed type/bug The issue is confirmed as a bug. labels May 16, 2023
@D3Hunter D3Hunter mentioned this issue May 16, 2023
Merged
12 tasks
@lance6716 lance6716 added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. labels May 16, 2023
ti-chi-bot bot pushed a commit that referenced this issue May 18, 2023
@D3Hunter
lightning: add send-kv-size to avoid oom when each kv is large on def…
ca62944 
…ault config (#43870)

close #43853
@ti-chi-bot ti-chi-bot mentioned this issue May 18, 2023
Closed
12 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue May 18, 2023
@D3Hunter @ti-chi-bot
lightning: add send-kv-size to avoid oom when each kv is large on def…
bb45f8b 
…ault config (pingcap#43870)

close pingcap#43853
@ti-chi-bot ti-chi-bot mentioned this issue May 18, 2023
Open
12 tasks
@D3Hunter D3Hunter mentioned this issue Jun 7, 2023
Merged
12 tasks
@D3Hunter D3Hunter removed affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. labels Jun 7, 2023
ti-chi-bot bot pushed a commit that referenced this issue Jun 7, 2023
@D3Hunter
lightning: deprecate send-kv-pairs (#44463)
bc45bdf 
ref #43853
@GMHDBJD GMHDBJD mentioned this issue Jun 25, 2023
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/lightning This issue is related to Lightning of TiDB. severity/moderate type/feature-request Categorizes issue or PR as related to a new feature.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

lightning: add send-kv-size to avoid oom when each kv is large on default config pingcap/tidb
2 participants
@lance6716 @D3Hunter