Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve] Speed up ClickhouseFile Local generate a mmap object #5822

Merged
merged 3 commits into from
Nov 16, 2023

Conversation

viverlxl
Copy link
Contributor

@viverlxl viverlxl commented Nov 9, 2023

Purpose of this pull request

Speed up ClickhouseFile Local generate data_local.log file

Does this PR introduce any user-facing change?

No

How was this patch tested?

Use the previous test case, I had test in my production env, eg: #5819

Check list

@viverlxl viverlxl changed the title fix-5819: 优化CK File模式写Local mmap 创建频繁,导致写入慢 fix-5819: sink CK File Local model A row generate a mmap object, write data slowly Nov 9, 2023
@viverlxl viverlxl changed the title fix-5819: sink CK File Local model A row generate a mmap object, write data slowly fix-5819: Sink CK File Local model A row generate a mmap object, write data slowly Nov 9, 2023
@viverlxl viverlxl changed the title fix-5819: Sink CK File Local model A row generate a mmap object, write data slowly [Fix]: Sink CK File Local model A row generate a mmap object, write data slowly issue: 5819 Nov 9, 2023
@Hisoka-X
Copy link
Member

Hisoka-X commented Nov 9, 2023

Thanks @viverlxl ! Please enable the test case run on your fork repository. You can follow the guide.
image

@Hisoka-X Hisoka-X changed the title [Fix]: Sink CK File Local model A row generate a mmap object, write data slowly issue: 5819 [Improve] Speed up ClickhouseFile Local generate a mmap object Nov 9, 2023
@viverlxl
Copy link
Contributor Author

Thanks @viverlxl ! Please enable the test case run on your fork repository. You can follow the guide. image
Done

@Hisoka-X
Copy link
Member

Please add some empty commit to retrigger CI.

@Hisoka-X
Copy link
Member

Also I think you should add comment for other better know this PR doing what, and the reason of why we should add this PR?
image

@viverlxl
Copy link
Contributor Author

Also I think you should add comment for other better know this PR doing what, and the reason of why we should add this PR? image
done

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @viverlxl !

@Hisoka-X Hisoka-X linked an issue Nov 16, 2023 that may be closed by this pull request
3 tasks
@hailin0 hailin0 merged commit cf39e29 into apache:dev Nov 16, 2023
4 checks passed
@viverlxl viverlxl deleted the fix-5819 branch November 16, 2023 07:30
alextinng pushed a commit to alextinng/seatunnel that referenced this pull request Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] [Module Name] Hive2CK using CK File generate data_local.log slow
3 participants