Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](index) add more inverted index profile metrics #36696

Merged
merged 4 commits into from
Aug 5, 2024
Merged

[opt](index) add more inverted index profile metrics #36696

merged 4 commits into from
Aug 5, 2024

Conversation

xiaokang
Copy link
Contributor

@xiaokang xiaokang commented Jun 22, 2024 •
edited
Loading

add more inverted index query profile:

  • InvertedIndexQueryNullBitmapTime
  • InvertedIndexSearcherCacheHit
  • InvertedIndexSearcherCacheMiss

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xiaokang
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40026 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 33b89f60d90f9278e46d1a7b8da000d4d6477ab2, data reload: false

------ Round 1 ----------------------------------
q1	17614	4459	4460	4459
q2	2006	193	194	193
q3	10451	1157	1076	1076
q4	10201	795	859	795
q5	7492	2718	2603	2603
q6	227	138	135	135
q7	966	616	613	613
q8	9214	2091	2109	2091
q9	8911	6498	6485	6485
q10	8936	3719	3727	3719
q11	468	240	236	236
q12	464	240	234	234
q13	18933	2990	2988	2988
q14	276	236	220	220
q15	530	487	484	484
q16	543	384	384	384
q17	973	614	730	614
q18	8092	7518	7356	7356
q19	6286	1404	1488	1404
q20	660	315	340	315
q21	4933	3277	3911	3277
q22	413	345	346	345
Total cold run time: 118589 ms
Total hot run time: 40026 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4437	4289	4315	4289
q2	373	280	283	280
q3	3103	2872	2926	2872
q4	1986	1728	1819	1728
q5	5572	5564	5506	5506
q6	228	133	142	133
q7	2267	1861	1895	1861
q8	3323	3465	3443	3443
q9	8782	8834	8802	8802
q10	4164	3722	3834	3722
q11	594	490	496	490
q12	830	684	632	632
q13	16099	3204	3205	3204
q14	304	275	270	270
q15	511	525	471	471
q16	505	428	433	428
q17	1821	1522	1503	1503
q18	8114	8033	7759	7759
q19	1836	1456	1596	1456
q20	2155	1849	1849	1849
q21	9326	4958	4793	4793
q22	646	556	551	551
Total cold run time: 76976 ms
Total hot run time: 56042 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170309 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 33b89f60d90f9278e46d1a7b8da000d4d6477ab2, data reload: false

query1	915	378	380	378
query2	6445	2378	2343	2343
query3	6631	209	215	209
query4	19659	17350	17324	17324
query5	3575	471	468	468
query6	239	172	175	172
query7	4514	294	301	294
query8	322	306	287	287
query9	8084	2442	2439	2439
query10	546	300	278	278
query11	10531	10056	9908	9908
query12	120	93	85	85
query13	1571	360	360	360
query14	10249	6179	7732	6179
query15	236	192	188	188
query16	7719	259	260	259
query17	1917	531	514	514
query18	1948	266	272	266
query19	199	158	152	152
query20	93	85	80	80
query21	214	134	129	129
query22	4354	4232	4139	4139
query23	33838	33627	33615	33615
query24	10767	2925	2937	2925
query25	597	386	374	374
query26	719	155	155	155
query27	2374	340	328	328
query28	6063	2172	2156	2156
query29	883	633	624	624
query30	242	152	158	152
query31	965	759	793	759
query32	100	52	54	52
query33	641	287	284	284
query34	916	493	488	488
query35	782	649	666	649
query36	1147	952	981	952
query37	143	76	72	72
query38	2917	2849	2811	2811
query39	931	822	859	822
query40	228	128	136	128
query41	55	54	55	54
query42	115	106	107	106
query43	614	538	556	538
query44	1091	724	743	724
query45	192	165	167	165
query46	1086	731	717	717
query47	1859	1757	1767	1757
query48	363	299	299	299
query49	850	409	420	409
query50	764	389	393	389
query51	6921	6734	6780	6734
query52	99	98	92	92
query53	358	292	292	292
query54	833	446	453	446
query55	76	74	74	74
query56	287	266	276	266
query57	1155	1059	1068	1059
query58	246	240	251	240
query59	3491	3275	3077	3077
query60	310	282	278	278
query61	94	87	87	87
query62	607	454	449	449
query63	322	299	291	291
query64	8460	2186	1709	1709
query65	3183	3092	3122	3092
query66	739	326	331	326
query67	15437	15031	14946	14946
query68	4584	543	547	543
query69	643	456	454	454
query70	1209	1072	1118	1072
query71	383	282	273	273
query72	6967	2760	2567	2567
query73	756	322	321	321
query74	5918	5467	5486	5467
query75	3471	2681	2705	2681
query76	2762	987	863	863
query77	622	307	316	307
query78	10403	9815	9696	9696
query79	2421	513	521	513
query80	1336	475	455	455
query81	558	229	224	224
query82	953	108	113	108
query83	263	175	167	167
query84	256	87	84	84
query85	1332	276	276	276
query86	441	296	287	287
query87	3282	3087	3077	3077
query88	3910	2357	2348	2348
query89	473	396	390	390
query90	1690	185	190	185
query91	130	100	99	99
query92	66	50	52	50
query93	2342	515	510	510
query94	1039	189	185	185
query95	421	311	314	311
query96	588	263	260	260
query97	3217	3074	3071	3071
query98	230	203	199	199
query99	1258	853	857	853
Total cold run time: 267865 ms
Total hot run time: 170309 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 33b89f60d90f9278e46d1a7b8da000d4d6477ab2, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.05
query3	0.22	0.05	0.06
query4	1.67	0.07	0.07
query5	0.49	0.48	0.49
query6	1.14	0.73	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.48	0.51
query10	0.54	0.56	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.78	0.79
query15	0.87	0.81	0.81
query16	0.35	0.37	0.38
query17	1.02	1.04	1.01
query18	0.22	0.22	0.25
query19	1.91	1.73	1.73
query20	0.01	0.01	0.01
query21	15.43	0.68	0.68
query22	4.02	6.96	2.34
query23	18.24	1.37	1.31
query24	2.10	0.26	0.22
query25	0.15	0.09	0.09
query26	0.28	0.19	0.17
query27	0.09	0.08	0.09
query28	13.17	1.02	0.99
query29	12.67	3.28	3.21
query30	0.26	0.06	0.06
query31	2.86	0.38	0.38
query32	3.26	0.48	0.47
query33	2.88	2.92	2.90
query34	17.03	4.43	4.36
query35	4.45	4.57	4.48
query36	0.65	0.46	0.46
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.15	0.14
query41	0.11	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.14 s
Total hot run time: 31.04 s

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

gavinchou
gavinchou previously approved these changes Jun 22, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@xiaokang xiaokang force-pushed the add_index_profile_master branch from ec6d42e to bdd5da7 Compare July 8, 2024 03:18
@xiaokang
Copy link
Contributor Author

xiaokang commented Jul 8, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 8, 2024
@xiaokang xiaokang added approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.x dev/2.0.12-merged labels Jul 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40645 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bdd5da770fd0fb94d553e5917be63cc552d68d16, data reload: false

------ Round 1 ----------------------------------
q1	17649	4741	4451	4451
q2	2035	203	195	195
q3	10438	1232	1012	1012
q4	10197	843	760	760
q5	7484	2679	2583	2583
q6	224	142	139	139
q7	974	606	608	606
q8	9244	2111	2139	2111
q9	9189	6545	6513	6513
q10	9009	3686	3749	3686
q11	454	257	236	236
q12	465	242	229	229
q13	17763	2995	3039	2995
q14	271	224	237	224
q15	529	494	474	474
q16	538	392	372	372
q17	996	694	691	691
q18	7954	7461	7474	7461
q19	7297	1541	1374	1374
q20	673	317	334	317
q21	4919	3880	3972	3880
q22	399	338	336	336
Total cold run time: 118701 ms
Total hot run time: 40645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4446	4316	4292	4292
q2	368	261	276	261
q3	3164	2878	2891	2878
q4	2026	1738	1706	1706
q5	5514	5489	5515	5489
q6	236	135	139	135
q7	2282	1831	1849	1831
q8	3294	3486	3456	3456
q9	8732	8855	8732	8732
q10	4089	3759	3909	3759
q11	578	495	503	495
q12	843	652	611	611
q13	15867	3148	3177	3148
q14	303	292	270	270
q15	531	483	483	483
q16	492	457	437	437
q17	1831	1553	1515	1515
q18	8189	8020	7793	7793
q19	1870	1676	1635	1635
q20	2164	1885	1863	1863
q21	7143	4865	4821	4821
q22	619	556	573	556
Total cold run time: 74581 ms
Total hot run time: 56166 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174142 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bdd5da770fd0fb94d553e5917be63cc552d68d16, data reload: false

query1	926	381	365	365
query2	6453	2418	2409	2409
query3	6651	207	212	207
query4	28144	17464	17333	17333
query5	3586	476	478	476
query6	279	163	176	163
query7	4594	296	294	294
query8	325	300	313	300
query9	8487	2410	2382	2382
query10	579	321	281	281
query11	11830	10117	10212	10117
query12	119	85	86	85
query13	1656	377	361	361
query14	10235	6881	7938	6881
query15	222	182	182	182
query16	7749	309	297	297
query17	1829	541	537	537
query18	1943	278	270	270
query19	197	152	155	152
query20	88	82	85	82
query21	212	130	129	129
query22	4361	4038	4092	4038
query23	34009	33658	33672	33658
query24	10853	2923	2839	2839
query25	576	393	409	393
query26	717	161	152	152
query27	2275	319	327	319
query28	5726	2166	2117	2117
query29	900	634	655	634
query30	261	155	149	149
query31	967	768	768	768
query32	100	56	56	56
query33	643	299	298	298
query34	890	503	495	495
query35	736	638	638	638
query36	1120	992	998	992
query37	140	77	84	77
query38	2939	2868	2789	2789
query39	883	851	786	786
query40	205	126	127	126
query41	51	51	51	51
query42	115	95	99	95
query43	597	549	569	549
query44	1072	727	731	727
query45	196	159	161	159
query46	1072	742	710	710
query47	1837	1758	1746	1746
query48	372	307	302	302
query49	840	407	401	401
query50	771	394	392	392
query51	6848	6831	6797	6797
query52	106	92	92	92
query53	377	292	291	291
query54	845	452	449	449
query55	76	74	75	74
query56	307	284	295	284
query57	1118	1063	1048	1048
query58	249	269	253	253
query59	3528	3252	3371	3252
query60	317	300	322	300
query61	117	114	117	114
query62	592	456	450	450
query63	337	297	290	290
query64	9250	2213	1607	1607
query65	3146	3092	3115	3092
query66	698	332	335	332
query67	15641	15237	15154	15154
query68	5744	531	537	531
query69	703	411	356	356
query70	1158	1148	1101	1101
query71	439	283	279	279
query72	8716	5740	5926	5740
query73	778	329	320	320
query74	5965	5469	5502	5469
query75	4240	2723	2683	2683
query76	3507	966	945	945
query77	676	302	293	293
query78	9647	8983	10019	8983
query79	2914	515	517	515
query80	2403	463	469	463
query81	582	222	226	222
query82	637	108	107	107
query83	279	167	166	166
query84	273	88	89	88
query85	1286	299	303	299
query86	446	322	331	322
query87	3273	3095	3096	3095
query88	4036	2477	2475	2475
query89	484	387	389	387
query90	1894	195	188	188
query91	132	103	103	103
query92	66	48	49	48
query93	3327	511	497	497
query94	1195	268	208	208
query95	403	316	319	316
query96	595	269	271	269
query97	3189	3043	2996	2996
query98	229	203	201	201
query99	1242	839	857	839
Total cold run time: 285338 ms
Total hot run time: 174142 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bdd5da770fd0fb94d553e5917be63cc552d68d16, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.65	0.08	0.08
query5	0.50	0.50	0.49
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.56	0.50	0.50
query10	0.55	0.55	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.58	0.58	0.58
query14	0.78	0.78	0.77
query15	0.84	0.82	0.80
query16	0.36	0.36	0.36
query17	0.95	1.03	1.04
query18	0.22	0.24	0.25
query19	1.73	1.72	1.83
query20	0.01	0.01	0.01
query21	15.40	0.76	0.65
query22	4.03	6.75	2.21
query23	18.22	1.35	1.19
query24	2.14	0.23	0.23
query25	0.15	0.08	0.08
query26	0.30	0.19	0.20
query27	0.46	0.24	0.22
query28	13.18	1.03	1.00
query29	12.58	3.27	3.29
query30	0.26	0.06	0.05
query31	2.87	0.38	0.39
query32	3.27	0.47	0.46
query33	2.87	2.85	2.87
query34	17.19	4.31	4.38
query35	4.41	4.40	4.43
query36	0.66	0.45	0.48
query37	0.19	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.32 s
Total hot run time: 30.76 s

@xiaokang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40088 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7ed5d91221ee47a65a168a0a82bd07fefc259742, data reload: false

------ Round 1 ----------------------------------
q1	17604	4416	4308	4308
q2	2010	196	189	189
q3	10504	1215	1139	1139
q4	10182	710	863	710
q5	7560	2762	2703	2703
q6	220	142	141	141
q7	989	601	605	601
q8	9211	2101	2094	2094
q9	8834	6558	6551	6551
q10	8892	3824	3812	3812
q11	473	235	243	235
q12	403	235	226	226
q13	18561	2986	3001	2986
q14	274	229	236	229
q15	523	484	481	481
q16	508	379	372	372
q17	984	596	695	596
q18	8133	7470	7403	7403
q19	6403	1542	1489	1489
q20	704	334	339	334
q21	4975	3154	3168	3154
q22	383	335	342	335
Total cold run time: 118330 ms
Total hot run time: 40088 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4446	4246	4259	4246
q2	390	272	262	262
q3	3050	2938	2897	2897
q4	1962	1656	1769	1656
q5	5664	5589	5495	5495
q6	223	139	139	139
q7	2202	1917	1858	1858
q8	3277	3431	3489	3431
q9	8788	8886	8901	8886
q10	4097	3818	3785	3785
q11	601	504	509	504
q12	815	649	650	649
q13	16011	3169	3187	3169
q14	333	305	290	290
q15	532	497	497	497
q16	492	430	445	430
q17	1832	1518	1505	1505
q18	8074	8238	8022	8022
q19	1843	1670	1542	1542
q20	2206	1898	1879	1879
q21	5019	4819	4851	4819
q22	619	558	552	552
Total cold run time: 72476 ms
Total hot run time: 56513 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176219 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ed5d91221ee47a65a168a0a82bd07fefc259742, data reload: false

query1	929	374	362	362
query2	7575	2414	2467	2414
query3	6636	212	218	212
query4	27701	17415	17394	17394
query5	3688	525	485	485
query6	261	160	161	160
query7	4603	302	307	302
query8	324	297	301	297
query9	8481	2443	2441	2441
query10	449	291	284	284
query11	11891	10044	10087	10044
query12	118	90	80	80
query13	1640	365	360	360
query14	10581	7716	7738	7716
query15	243	185	188	185
query16	7581	314	312	312
query17	1363	534	555	534
query18	1840	278	276	276
query19	190	155	152	152
query20	93	81	82	81
query21	204	127	129	127
query22	4437	4094	4035	4035
query23	34199	33717	33699	33699
query24	11246	2911	2927	2911
query25	651	390	392	390
query26	1194	158	161	158
query27	2597	275	283	275
query28	7535	2142	2132	2132
query29	918	659	654	654
query30	255	151	153	151
query31	977	777	780	777
query32	100	55	59	55
query33	802	300	311	300
query34	996	512	520	512
query35	709	612	608	608
query36	1092	980	978	978
query37	159	91	85	85
query38	2907	2876	2863	2863
query39	926	864	848	848
query40	207	136	134	134
query41	59	53	50	50
query42	115	104	103	103
query43	606	570	569	569
query44	1217	716	734	716
query45	197	170	173	170
query46	1087	738	725	725
query47	1874	1739	1748	1739
query48	375	308	298	298
query49	876	443	442	442
query50	789	405	405	405
query51	6887	6679	6697	6679
query52	113	95	98	95
query53	372	303	310	303
query54	917	471	460	460
query55	79	78	77	77
query56	315	290	302	290
query57	1162	1024	1075	1024
query58	266	265	289	265
query59	3488	3223	3254	3223
query60	337	309	309	309
query61	122	116	120	116
query62	810	648	660	648
query63	324	296	304	296
query64	9587	2287	1753	1753
query65	3183	3156	3116	3116
query66	848	346	355	346
query67	15456	14933	15086	14933
query68	4617	537	551	537
query69	681	436	383	383
query70	1205	1132	1165	1132
query71	455	295	294	294
query72	9246	5768	5685	5685
query73	758	324	331	324
query74	5963	5551	5533	5533
query75	4130	2727	2705	2705
query76	3296	949	985	949
query77	701	326	308	308
query78	11550	9861	9144	9144
query79	11763	583	514	514
query80	949	465	483	465
query81	582	222	221	221
query82	771	133	131	131
query83	312	167	166	166
query84	283	87	87	87
query85	740	307	308	307
query86	376	313	326	313
query87	3307	3128	3076	3076
query88	5221	2476	2515	2476
query89	547	380	382	380
query90	2060	199	197	197
query91	131	104	102	102
query92	65	48	50	48
query93	6685	499	496	496
query94	1149	212	214	212
query95	419	323	333	323
query96	612	283	272	272
query97	3189	3025	3027	3025
query98	228	200	188	188
query99	1531	1293	1306	1293
Total cold run time: 303281 ms
Total hot run time: 176219 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7ed5d91221ee47a65a168a0a82bd07fefc259742, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.06
query4	1.68	0.09	0.08
query5	0.49	0.49	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.53	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.59
query14	0.75	0.78	0.81
query15	0.85	0.81	0.81
query16	0.37	0.37	0.35
query17	0.97	0.97	1.05
query18	0.22	0.21	0.22
query19	1.79	1.71	1.70
query20	0.02	0.01	0.01
query21	15.39	0.79	0.68
query22	4.63	6.56	2.15
query23	18.16	1.37	1.33
query24	2.18	0.22	0.23
query25	0.16	0.09	0.08
query26	0.31	0.21	0.21
query27	0.45	0.22	0.23
query28	13.26	1.03	1.00
query29	12.59	3.33	3.32
query30	0.26	0.06	0.06
query31	2.88	0.38	0.38
query32	3.28	0.48	0.47
query33	2.91	2.95	2.94
query34	16.92	4.32	4.29
query35	4.44	4.41	4.39
query36	0.65	0.47	0.46
query37	0.19	0.16	0.16
query38	0.15	0.16	0.15
query39	0.04	0.03	0.03
query40	0.16	0.12	0.14
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.85 s
Total hot run time: 30.91 s

@xiaokang
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40061 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c5a2a9cf202c2e31095c7c5559a642516afb0d5, data reload: false

------ Round 1 ----------------------------------
q1	17615	4480	4274	4274
q2	2023	190	183	183
q3	10531	1193	1128	1128
q4	10194	807	821	807
q5	7554	2679	2654	2654
q6	221	138	142	138
q7	963	612	592	592
q8	9208	2108	2127	2108
q9	8774	6585	6576	6576
q10	8813	3820	3848	3820
q11	455	241	239	239
q12	397	224	222	222
q13	17926	2977	3010	2977
q14	281	242	244	242
q15	528	481	481	481
q16	494	389	374	374
q17	984	678	678	678
q18	8237	7557	7397	7397
q19	9096	1417	1454	1417
q20	706	319	316	316
q21	4965	3158	3316	3158
q22	347	280	280	280
Total cold run time: 120312 ms
Total hot run time: 40061 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4265	4245	4245
q2	361	273	266	266
q3	3021	2739	2753	2739
q4	1877	1567	1640	1567
q5	5292	5365	5310	5310
q6	229	137	131	131
q7	2112	1732	1728	1728
q8	3215	3351	3312	3312
q9	8492	8424	8347	8347
q10	3918	3651	3731	3651
q11	564	496	506	496
q12	799	590	642	590
q13	17358	3033	3030	3030
q14	292	268	265	265
q15	513	469	475	469
q16	462	408	422	408
q17	1801	1484	1485	1484
q18	7602	7461	7633	7461
q19	1668	1472	1695	1472
q20	1973	1759	1775	1759
q21	4878	4768	4758	4758
q22	586	507	528	507
Total cold run time: 71396 ms
Total hot run time: 53995 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172547 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c5a2a9cf202c2e31095c7c5559a642516afb0d5, data reload: false

query1	930	383	366	366
query2	6453	1984	1881	1881
query3	6646	201	212	201
query4	28438	17641	17406	17406
query5	4228	486	478	478
query6	273	194	162	162
query7	4590	300	293	293
query8	254	202	191	191
query9	8414	2434	2422	2422
query10	452	295	280	280
query11	13415	9968	9942	9942
query12	132	86	87	86
query13	1646	372	359	359
query14	9811	7715	7896	7715
query15	247	168	167	167
query16	7528	324	319	319
query17	1775	556	598	556
query18	1331	271	266	266
query19	188	146	146	146
query20	91	82	82	82
query21	200	124	124	124
query22	4210	4184	4011	4011
query23	33703	33011	33208	33011
query24	11752	2955	2796	2796
query25	607	359	373	359
query26	1736	148	146	146
query27	2687	269	272	269
query28	7381	1999	1990	1990
query29	910	618	601	601
query30	284	146	149	146
query31	991	775	752	752
query32	101	51	52	51
query33	738	288	287	287
query34	894	497	478	478
query35	682	585	575	575
query36	1096	908	911	908
query37	136	79	79	79
query38	2813	2758	2691	2691
query39	856	812	832	812
query40	280	115	118	115
query41	45	46	44	44
query42	110	101	98	98
query43	516	461	470	461
query44	1262	722	721	721
query45	189	159	162	159
query46	1077	748	773	748
query47	1868	1766	1747	1747
query48	362	280	293	280
query49	1174	421	402	402
query50	780	393	381	381
query51	6910	6890	6740	6740
query52	104	87	90	87
query53	361	286	282	282
query54	838	441	439	439
query55	100	77	72	72
query56	282	259	266	259
query57	1146	1030	1048	1030
query58	262	236	249	236
query59	2867	2758	2544	2544
query60	295	271	277	271
query61	94	92	92	92
query62	853	650	651	650
query63	315	297	287	287
query64	10685	2175	7470	2175
query65	3150	3083	3092	3083
query66	1325	329	328	328
query67	15629	15030	14860	14860
query68	4654	520	537	520
query69	535	326	323	323
query70	1110	1087	1076	1076
query71	442	281	270	270
query72	8554	5505	5965	5505
query73	748	328	321	321
query74	6200	5639	5721	5639
query75	4094	2647	2718	2647
query76	3190	948	873	873
query77	683	295	306	295
query78	10728	8997	9531	8997
query79	2360	516	523	516
query80	2580	487	472	472
query81	597	223	219	219
query82	778	144	134	134
query83	315	168	169	168
query84	268	92	86	86
query85	2260	302	294	294
query86	477	301	309	301
query87	3401	3095	3115	3095
query88	4102	2382	2363	2363
query89	486	390	388	388
query90	2002	203	196	196
query91	133	101	99	99
query92	68	47	51	47
query93	2875	502	502	502
query94	1100	282	220	220
query95	409	321	316	316
query96	609	271	267	267
query97	3205	3004	2979	2979
query98	212	196	193	193
query99	1516	1231	1272	1231
Total cold run time: 292189 ms
Total hot run time: 172547 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c5a2a9cf202c2e31095c7c5559a642516afb0d5, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.69	0.08	0.06
query5	0.49	0.47	0.50
query6	1.14	0.72	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.55	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.78	0.77	0.79
query15	0.85	0.80	0.81
query16	0.37	0.37	0.37
query17	0.96	0.94	0.95
query18	0.23	0.22	0.21
query19	1.83	1.72	1.76
query20	0.01	0.01	0.01
query21	15.44	0.74	0.67
query22	4.25	7.01	2.20
query23	18.31	1.45	1.27
query24	1.88	0.28	0.22
query25	0.16	0.10	0.09
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.26	1.04	1.00
query29	12.61	3.35	3.33
query30	0.25	0.07	0.06
query31	2.87	0.39	0.41
query32	3.23	0.48	0.47
query33	2.86	2.94	2.94
query34	17.01	4.36	4.40
query35	4.38	4.40	4.42
query36	0.65	0.49	0.50
query37	0.19	0.16	0.16
query38	0.16	0.16	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.38 s
Total hot run time: 30.99 s

airborne12
airborne12 approved these changes Aug 5, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

zzzxl1993
zzzxl1993 approved these changes Aug 5, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit 6942e16 into apache:master Aug 5, 2024
26 of 29 checks passed
xiaokang added a commit to xiaokang/doris that referenced this pull request Aug 5, 2024
@xiaokang
[opt](index) add more inverted index profile metrics (apache#36696)
583d0e7 
add more inverted index query profile:
- InvertedIndexQueryNullBitmapTime
- InvertedIndexSearcherCacheHit
- InvertedIndexSearcherCacheMiss
@xiaokang xiaokang mentioned this pull request Aug 5, 2024
Merged
@xiaokang xiaokang removed the dev/3.0.x label Aug 7, 2024
xiaokang added a commit that referenced this pull request Aug 8, 2024
@xiaokang
[opt](index) add more inverted index profile metrics #36696 (#38858)
44cb797
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@xiaokang @dataroaring
[opt](index) add more inverted index profile metrics (#36696)
a7e7b73 
add more inverted index query profile:
- InvertedIndexQueryNullBitmapTime
- InvertedIndexSearcherCacheHit
- InvertedIndexSearcherCacheMiss
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@xiaokang @dataroaring
[opt](index) add more inverted index profile metrics (#36696)
2ffa016 
add more inverted index query profile:
- InvertedIndexQueryNullBitmapTime
- InvertedIndexSearcherCacheHit
- InvertedIndexSearcherCacheMiss
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@airborne12 airborne12 airborne12 approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

@gavinchou gavinchou gavinchou left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.12-merged dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@xiaokang @doris-robot @airborne12 @gavinchou @zzzxl1993 @dataroaring