Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement sort index hints for postgres eventstore #2204

Merged
merged 1 commit into from
Jan 18, 2022

Conversation

IhostVlad
Copy link
Contributor

~12 million events eventstore, RDS with 32 ACU
Speed up ~40 times within specific eventstore

Before changes: execution time ~34000 ms, execution plan:

Limit (cost=2529306.86..2529309.36 rows=1000 width=237) (actual time=34616.751..34617.251 rows=1000 loops=1)
-> Sort (cost=2529306.86..2529946.86 rows=256000 width=237) (actual time=34616.750..34617.182 rows=1000 loops=1)
Sort Key: events."timestamp", events."threadCounter", events."threadId"
Sort Method: top-N heapsort Memory: 1926kB
-> Append (cost=556.04..2515270.65 rows=256000 width=237) (actual time=18.115..34592.014 rows=197739 loops=1)
-> Limit (cost=556.04..10045.03 rows=1000 width=237) (actual time=18.114..45.263 rows=1000 loops=1)
-> Bitmap Heap Scan on events (cost=556.04..48703.20 rows=5074 width=237) (actual time=18.113..45.180 rows=1000 loops=1)
Recheck Cond: (("threadId" = 0) AND ("threadCounter" >= '0'::bigint))
Filter: ((type)::text = ANY ('{UPLOAD_CREATED,REPORT_TASK_STARTED}'::text[]))
Rows Removed by Filter: 1150
Heap Blocks: exact=1158
-> Bitmap Index Scan on events_pkey (cost=0.00..554.77 rows=14221 width=0) (actual time=15.769..15.769 rows=8402 loops=1)
Index Cond: (("threadId" = 0) AND ("threadCounter" >= '0'::bigint))
-> Limit (cost=556.04..10045.03 rows=1000 width=237) (actual time=16.213..44.080 rows=1000 loops=1)
-> Bitmap Heap Scan on events events_1 (cost=556.04..48703.20 rows=5074 width=237) (actual time=16.211..43.999 rows=1000 loops=1)
Recheck Cond: (("threadId" = 1) AND ("threadCounter" >= '0'::bigint))
Filter: ((type)::text = ANY ('{UPLOAD_CREATED,REPORT_TASK_STARTED}'::text[]))
Rows Removed by Filter: 893
Heap Blocks: exact=1232
-> Bitmap Index Scan on events_pkey (cost=0.00..554.77 rows=14221 width=0) (actual time=14.652..14.652 rows=2072 loops=1)
.......

After changes: execution time ~700 ms, execution plan:

Sort (cost=2788858.58..2788861.08 rows=1000 width=237) (actual time=293.167..293.437 rows=1000 loops=1)
Sort Key: events."threadCounter", events."threadId"
Sort Method: quicksort Memory: 1613kB
-> Limit (cost=2788796.25..2788798.75 rows=1000 width=237) (actual time=292.061..292.418 rows=1000 loops=1)
-> Sort (cost=2788796.25..2789436.25 rows=256000 width=237) (actual time=292.060..292.349 rows=1000 loops=1)
Sort Key: events."timestamp"
Sort Method: top-N heapsort Memory: 1926kB
-> Append (cost=0.56..2774760.04 rows=256000 width=237) (actual time=0.041..268.203 rows=197739 loops=1)
-> Limit (cost=0.56..11136.97 rows=1000 width=237) (actual time=0.040..1.421 rows=1000 loops=1)
-> Index Scan using events_pkey on events (cost=0.56..56506.68 rows=5074 width=237) (actual time=0.040..1.350 rows=1000 loops=1)
Index Cond: (("threadId" = 0) AND ("threadCounter" >= '0'::bigint))
Filter: ((type)::text = ANY ('{UPLOAD_CREATED,REPORT_TASK_STARTED}'::text[]))
Rows Removed by Filter: 1150
-> Limit (cost=0.56..11136.97 rows=1000 width=237) (actual time=0.027..1.291 rows=1000 loops=1)
-> Index Scan using events_pkey on events events_1 (cost=0.56..56506.68 rows=5074 width=237) (actual time=0.026..1.220 rows=1000 loops=1)
Index Cond: (("threadId" = 1) AND ("threadCounter" >= '0'::bigint))
Filter: ((type)::text = ANY ('{UPLOAD_CREATED,REPORT_TASK_STARTED}'::text[]))
Rows Removed by Filter: 893
-> Limit (cost=0.56..11136.97 rows=1000 width=237) (actual time=0.025..0.458 rows=388 loops=1)
-> Index Scan using events_pkey on events events_2 (cost=0.56..56506.68 rows=5074 width=237) (actual time=0.024..0.431 rows=388 loops=1)
.......

@IhostVlad IhostVlad force-pushed the feature/es-pg-sort-hints branch from bcfde5c to 0b14f87 Compare January 18, 2022 10:27
@FreeSlave FreeSlave merged commit 5a1aea9 into dev Jan 18, 2022
@FreeSlave FreeSlave added the enhancement Issues related to potential improvements and new features label Jan 18, 2022
@FreeSlave FreeSlave deleted the feature/es-pg-sort-hints branch January 18, 2022 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Issues related to potential improvements and new features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants