Optimized batch processing #10131

CyberROFL · 2024-10-05T15:51:09Z

Changelog entry

Optimized batch processing in Topics.

Changelog category

Performance improvement

Additional information

...

github-actions · 2024-10-05T15:55:00Z

⚪ 2024-10-05 15:54:59 UTC Pre-commit check linux-x86_64-release-asan for bfcdd87 has started.
⚪ 2024-10-05 15:55:10 UTC Artifacts will be uploaded here
⚪ 2024-10-05 15:58:02 UTC ya make is running...
⚫ 2024-10-05 16:10:53 UTC Check cancelled

github-actions · 2024-10-05T15:55:28Z

⚪ 2024-10-05 15:55:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for bfcdd87 has started.
⚪ 2024-10-05 15:55:38 UTC Artifacts will be uploaded here
⚪ 2024-10-05 15:58:35 UTC ya make is running...
⚫ 2024-10-05 16:10:56 UTC Check cancelled

github-actions · 2024-10-05T16:14:47Z

⚪ 2024-10-05 16:14:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for af44f80 has started.
⚪ 2024-10-05 16:14:57 UTC Artifacts will be uploaded here
⚪ 2024-10-05 16:17:49 UTC ya make is running...
🟡 2024-10-05 17:07:03 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16491	14977	0	12	1403	99

⚪ 2024-10-05 17:08:27 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-05 17:20:02 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
134 (only retried tests)	18	0	8	14	94

⚪ 2024-10-05 17:20:11 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-05 17:31:44 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
116 (only retried tests)	1	0	8	14	93

🟢 2024-10-05 17:31:51 UTC Build successful.
🟢 2024-10-05 17:32:12 UTC ydbd size 2.8 GiB changed* by -22.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `54b866e`	merge: `af44f80`	diff	diff %
ydbd size	3 002 276 928 Bytes	3 002 254 016 Bytes	-22.4 KiB	-0.001%
ydbd stripped size	477 418 232 Bytes	477 417 912 Bytes	-320 Bytes	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-05T16:14:53Z

⚪ 2024-10-05 16:14:52 UTC Pre-commit check linux-x86_64-release-asan for af44f80 has started.
⚪ 2024-10-05 16:15:03 UTC Artifacts will be uploaded here
⚪ 2024-10-05 16:18:01 UTC ya make is running...
🟡 2024-10-05 17:44:59 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11541	11307	0	74	58	102

🟢 2024-10-05 17:45:59 UTC Build successful.
🟢 2024-10-05 17:46:33 UTC ydbd size 5.7 GiB changed* by -36.8 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `54b866e`	merge: `af44f80`	diff	diff %
ydbd size	6 084 430 152 Bytes	6 084 392 464 Bytes	-36.8 KiB	-0.001%
ydbd stripped size	1 520 399 600 Bytes	1 520 404 848 Bytes	+5.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-05T18:34:36Z

⚪ 2024-10-05 18:34:36 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4c8f6d7 has started.
⚪ 2024-10-05 18:34:46 UTC Artifacts will be uploaded here
⚪ 2024-10-05 18:37:42 UTC ya make is running...
🟡 2024-10-05 19:29:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16492	14995	0	8	1390	99

⚪ 2024-10-05 19:31:17 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-05 19:42:32 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
115 (only retried tests)	20	0	1	0	94

⚪ 2024-10-05 19:42:40 UTC ya make is running... (failed tests rerun, try 3)
🟢 2024-10-05 19:53:11 UTC ydbd size 2.8 GiB changed* by -22.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `54b866e`	merge: `4c8f6d7`	diff	diff %
ydbd size	3 002 276 928 Bytes	3 002 254 016 Bytes	-22.4 KiB	-0.001%
ydbd stripped size	477 418 232 Bytes	477 417 912 Bytes	-320 Bytes	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2024-10-05 19:53:13 UTC Check cancelled

github-actions · 2024-10-05T18:34:40Z

⚪ 2024-10-05 18:34:39 UTC Pre-commit check linux-x86_64-release-asan for 4c8f6d7 has started.
⚪ 2024-10-05 18:34:49 UTC Artifacts will be uploaded here
⚪ 2024-10-05 18:37:39 UTC ya make is running...
⚫ 2024-10-05 19:52:48 UTC Check cancelled

github-actions · 2024-10-05T19:56:28Z

⚪ 2024-10-05 19:56:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for ed6c190 has started.
⚪ 2024-10-05 19:56:37 UTC Artifacts will be uploaded here
⚪ 2024-10-05 19:59:27 UTC ya make is running...
🟡 2024-10-05 20:46:25 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16490	15011	0	3	1377	99

⚪ 2024-10-05 20:47:45 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-05 20:58:54 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
99 (only retried tests)	5	0	1	0	93

⚪ 2024-10-05 20:59:02 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-05 21:10:11 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
94 (only retried tests)	0	0	1	0	93

🟢 2024-10-05 21:10:17 UTC Build successful.
🟢 2024-10-05 21:10:39 UTC ydbd size 2.8 GiB changed* by -22.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `54b866e`	merge: `ed6c190`	diff	diff %
ydbd size	3 002 276 928 Bytes	3 002 254 016 Bytes	-22.4 KiB	-0.001%
ydbd stripped size	477 418 232 Bytes	477 417 912 Bytes	-320 Bytes	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-05T19:57:00Z

⚪ 2024-10-05 19:57:00 UTC Pre-commit check linux-x86_64-release-asan for ed6c190 has started.
⚪ 2024-10-05 19:57:11 UTC Artifacts will be uploaded here
⚪ 2024-10-05 20:00:08 UTC ya make is running...
🟡 2024-10-05 21:27:15 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11536	11314	0	61	54	107

🟢 2024-10-05 21:28:14 UTC Build successful.
🟢 2024-10-05 21:28:48 UTC ydbd size 5.7 GiB changed* by -36.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `54b866e`	merge: `ed6c190`	diff	diff %
ydbd size	6 084 430 152 Bytes	6 084 392 768 Bytes	-36.5 KiB	-0.001%
ydbd stripped size	1 520 399 600 Bytes	1 520 404 912 Bytes	+5.2 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-05T21:43:14Z

⚪ 2024-10-05 21:43:13 UTC Pre-commit check linux-x86_64-relwithdebinfo for ed6c190 has started.
⚪ 2024-10-05 21:43:24 UTC Artifacts will be uploaded here
⚪ 2024-10-05 21:46:15 UTC ya make is running...
🟡 2024-10-05 22:35:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16489	15009	0	2	1378	100

⚪ 2024-10-05 22:36:25 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-05 22:47:35 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
101 (only retried tests)	7	0	0	0	94

🟢 2024-10-05 22:47:42 UTC Build successful.
🟢 2024-10-05 22:48:00 UTC ydbd size 2.8 GiB changed* by -22.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `54b866e`	merge: `ed6c190`	diff	diff %
ydbd size	3 002 276 928 Bytes	3 002 254 016 Bytes	-22.4 KiB	-0.001%
ydbd stripped size	477 418 232 Bytes	477 417 912 Bytes	-320 Bytes	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-05T21:43:30Z

⚪ 2024-10-05 21:43:29 UTC Pre-commit check linux-x86_64-release-asan for ed6c190 has started.
⚪ 2024-10-05 21:43:39 UTC Artifacts will be uploaded here
⚪ 2024-10-05 21:46:31 UTC ya make is running...
🟡 2024-10-05 23:10:00 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11537	11140	0	72	281	44

🟢 2024-10-05 23:10:58 UTC Build successful.
🟢 2024-10-05 23:11:30 UTC ydbd size 5.7 GiB changed* by -36.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `54b866e`	merge: `ed6c190`	diff	diff %
ydbd size	6 084 430 152 Bytes	6 084 392 768 Bytes	-36.5 KiB	-0.001%
ydbd stripped size	1 520 399 600 Bytes	1 520 404 912 Bytes	+5.2 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot added the new-feature label Oct 5, 2024

CyberROFL self-assigned this Oct 5, 2024

github-actions bot added performance and removed new-feature labels Oct 5, 2024

CyberROFL marked this pull request as ready for review October 5, 2024 15:52

CyberROFL requested a review from a team as a code owner October 5, 2024 15:52

CyberROFL requested a review from niksaveliev October 5, 2024 15:52

CyberROFL requested review from nshestakov and alexnick88 October 5, 2024 15:58

Optimized batch processing

72f52fa

CyberROFL force-pushed the pq-optimizations branch from 6ec4237 to 72f52fa Compare October 5, 2024 16:10

Fix

04cd3e0

CyberROFL requested a review from Alek5andr-Kotov October 5, 2024 18:36

Fix test

3933014

CyberROFL added the rebase-and-check Rebase PR with the current base branch and check label Oct 5, 2024

github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Oct 5, 2024

CyberROFL enabled auto-merge (squash) October 5, 2024 21:44

nshestakov approved these changes Oct 6, 2024

View reviewed changes

CyberROFL merged commit 791558d into ydb-platform:main Oct 6, 2024
13 of 14 checks passed

CyberROFL deleted the pq-optimizations branch October 6, 2024 08:13

GrigoriyPA mentioned this pull request Oct 8, 2024

YQ-3722 add simdjson parser into RD #10204

Merged

This was referenced Oct 21, 2024

Fix kafka cdc read and login without @ #10678

Merged

Allow short auto partitioning strategies yql #10701

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimized batch processing #10131

Optimized batch processing #10131

CyberROFL commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

Optimized batch processing #10131

Optimized batch processing #10131

Conversation

CyberROFL commented Oct 5, 2024 • edited Loading

Changelog entry

Changelog category

Additional information

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

github-actions bot commented Oct 5, 2024 • edited Loading

CyberROFL commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading

github-actions bot commented Oct 5, 2024 •

edited

Loading