Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable arena-based allocations for QueryService output rows #13174

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Enable arena-based allocations for QueryService output rows #13174

wants to merge 1 commit into from
+21 −15

Conversation

zinal
Copy link
Member

@zinal zinal commented Jan 4, 2025

Changelog entry

...

Changelog category

  • Performance improvement
  • Not for changelog (changelog entry is not required)

Additional information

...

@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 4, 2025 •
edited
Loading

⚪ 2025-01-04 17:46:28 UTC Pre-commit check linux-x86_64-relwithdebinfo for e0f3ad4 has started.
⚪ 2025-01-04 17:46:40 UTC Artifacts will be uploaded here
⚪ 2025-01-04 17:49:32 UTC ya make is running...
🟡 2025-01-04 18:36:15 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18696 17400 0 5 1169 122

⚪ 2025-01-04 18:37:52 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-04 18:49:18 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
185 (only retried tests) 64 0 1 0 120

⚪ 2025-01-04 18:49:26 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-04 19:00:53 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
176 (only retried tests) 56 0 0 0 120

🟢 2025-01-04 19:01:00 UTC Build successful.
🟢 2025-01-04 19:01:23 UTC ydbd size 2.1 GiB changed* by +18.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 9740792 merge: e0f3ad4 diff diff %
ydbd size 2 243 624 392 Bytes 2 243 642 880 Bytes +18.1 KiB +0.001%
ydbd stripped size 472 254 992 Bytes 472 255 440 Bytes +448 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@zinal
Copy link
Member Author

zinal commented Jan 4, 2025 •
edited
Loading

Trying to improve the performance of wide table scans through QueryService.
Initially checked on 24.3.13, flamegraphs before and after are attached.

flame-before
flame-after

The average speed of scans of 300k rows decreased from 11 seconds to about 7 seconds.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 4, 2025 •
edited
Loading

⚪ 2025-01-04 17:49:01 UTC Pre-commit check linux-x86_64-release-asan for e0f3ad4 has started.
⚪ 2025-01-04 17:49:13 UTC Artifacts will be uploaded here
⚪ 2025-01-04 17:52:11 UTC ya make is running...
🟡 2025-01-04 18:52:30 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11583 11512 0 38 4 29

⚪ 2025-01-04 18:53:28 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-04 19:06:26 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
120 (only retried tests) 84 0 8 2 26

⚪ 2025-01-04 19:06:35 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-04 19:18:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
64 (only retried tests) 35 0 2 0 27

🟢 2025-01-04 19:18:30 UTC Build successful.
🟢 2025-01-04 19:19:00 UTC ydbd size 3.6 GiB changed* by +28.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 9740792 merge: e0f3ad4 diff diff %
ydbd size 3 872 732 616 Bytes 3 872 762 240 Bytes +28.9 KiB +0.001%
ydbd stripped size 1 350 439 952 Bytes 1 350 443 088 Bytes +3.1 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

zinal added a commit to zinal/ydb-flamez-temp that referenced this pull request Jan 4, 2025
@zinal
ydb-platform/ydb#13174 files
aae5581
@zinal
Copy link
Member Author

zinal commented Jan 5, 2025

Statistics from main show 17% average runtime improvements for simple queries (index range scan + row fetch, 150k output rows per query).

Before changes

(base) $ ./ydb-selector1.sh selector1.xml selector0.sql 
2025-01-05 15:25:23 INFO  Selector1:93 - Reading configuration from selector1.xml...
2025-01-05 15:25:23 INFO  Selector1:95 - Reading the query file selector0.sql...
2025-01-05 15:25:23 INFO  Selector1:98 - Obtaining the JDBC connection...
янв. 05, 2025 3:25:23 PM tech.ydb.jdbc.YdbDriver register
INFO: YDB JDBC Driver registered: YDB JDBC Driver 2.3.6(based on SDK 2.3.6)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2025-01-05 15:25:23 INFO  Selector1:102 - Connected in 670 msec.
2025-01-05 15:25:23 INFO  Selector1:104 - Creating the worker instance...
2025-01-05 15:25:23 INFO  Selector1:106 - Entering the selector1 algorithm...
2025-01-05 15:25:28 INFO  Selector1:53 - step 0 count 150000: exec 4906 fetch 146
2025-01-05 15:25:33 INFO  Selector1:53 - step 1 count 150000: exec 4678 fetch 95
2025-01-05 15:25:38 INFO  Selector1:53 - step 2 count 150000: exec 4580 fetch 86
2025-01-05 15:25:43 INFO  Selector1:53 - step 3 count 150000: exec 4919 fetch 87
2025-01-05 15:25:48 INFO  Selector1:53 - step 4 count 150000: exec 4888 fetch 88
2025-01-05 15:25:53 INFO  Selector1:53 - step 5 count 150000: exec 4825 fetch 86
2025-01-05 15:25:58 INFO  Selector1:53 - step 6 count 150000: exec 4863 fetch 85
2025-01-05 15:26:03 INFO  Selector1:53 - step 7 count 150000: exec 4761 fetch 86
2025-01-05 15:26:08 INFO  Selector1:53 - step 8 count 150000: exec 4890 fetch 84
2025-01-05 15:26:12 INFO  Selector1:53 - step 9 count 150000: exec 4837 fetch 86
2025-01-05 15:26:13 INFO  Selector1:109 - Completed.

After changes

(base) $ ./ydb-selector1.sh selector1.xml selector0.sql 
2025-01-05 15:33:53 INFO  Selector1:93 - Reading configuration from selector1.xml...
2025-01-05 15:33:53 INFO  Selector1:95 - Reading the query file selector0.sql...
2025-01-05 15:33:53 INFO  Selector1:98 - Obtaining the JDBC connection...
янв. 05, 2025 3:33:53 PM tech.ydb.jdbc.YdbDriver register
INFO: YDB JDBC Driver registered: YDB JDBC Driver 2.3.6(based on SDK 2.3.6)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2025-01-05 15:33:54 INFO  Selector1:102 - Connected in 652 msec.
2025-01-05 15:33:54 INFO  Selector1:104 - Creating the worker instance...
2025-01-05 15:33:54 INFO  Selector1:106 - Entering the selector1 algorithm...
2025-01-05 15:33:59 INFO  Selector1:53 - step 0 count 150000: exec 4496 fetch 132
2025-01-05 15:34:03 INFO  Selector1:53 - step 1 count 150000: exec 4071 fetch 126
2025-01-05 15:34:07 INFO  Selector1:53 - step 2 count 150000: exec 3988 fetch 84
2025-01-05 15:34:11 INFO  Selector1:53 - step 3 count 150000: exec 4102 fetch 92
2025-01-05 15:34:15 INFO  Selector1:53 - step 4 count 150000: exec 3836 fetch 89
2025-01-05 15:34:19 INFO  Selector1:53 - step 5 count 150000: exec 3953 fetch 112
2025-01-05 15:34:23 INFO  Selector1:53 - step 6 count 150000: exec 3787 fetch 161
2025-01-05 15:34:27 INFO  Selector1:53 - step 7 count 150000: exec 3852 fetch 85
2025-01-05 15:34:31 INFO  Selector1:53 - step 8 count 150000: exec 3848 fetch 85
2025-01-05 15:34:35 INFO  Selector1:53 - step 9 count 150000: exec 3858 fetch 86
2025-01-05 15:34:35 INFO  Selector1:109 - Completed.

@zinal zinal mentioned this pull request Jan 5, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gridnevvvit gridnevvvit gridnevvvit approved these changes

Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zinal @gridnevvvit