Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](statistics)Clean expired TableStatsMeta. #39779

Merged
merged 1 commit into from
Aug 28, 2024
Merged

[improve](statistics)Clean expired TableStatsMeta. #39779

merged 1 commit into from
Aug 28, 2024

Conversation

Jibing-Li
Copy link
Contributor

Drop catalog and older version of drop database (before #39685), will not remove table stats object in memory. Need a background thread to clean the garbage.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review August 22, 2024 08:21
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37946 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b64e00b02a5dc3c860e6bf1db6d1c999cf249b8, data reload: false

------ Round 1 ----------------------------------
q1	18635	4524	4535	4524
q2	2875	183	177	177
q3	10799	1142	1150	1142
q4	10506	728	763	728
q5	7753	2914	2867	2867
q6	228	135	143	135
q7	974	619	605	605
q8	9506	2036	2021	2021
q9	7247	6541	6463	6463
q10	6998	2209	2186	2186
q11	447	239	247	239
q12	393	220	214	214
q13	17761	3016	3051	3016
q14	280	234	230	230
q15	522	478	495	478
q16	499	392	392	392
q17	957	660	706	660
q18	7352	6870	6905	6870
q19	1393	882	991	882
q20	672	330	335	330
q21	3881	2987	2785	2785
q22	1133	1029	1002	1002
Total cold run time: 110811 ms
Total hot run time: 37946 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4268	4254	4254
q2	384	277	273	273
q3	2879	2648	2640	2640
q4	1904	1677	1621	1621
q5	5413	5394	5359	5359
q6	220	129	127	127
q7	2104	1696	1686	1686
q8	3181	3320	3328	3320
q9	8384	8405	8393	8393
q10	3448	3215	3192	3192
q11	614	506	500	500
q12	776	621	628	621
q13	8697	3051	3043	3043
q14	312	278	275	275
q15	539	476	480	476
q16	484	424	421	421
q17	1783	1485	1485	1485
q18	7736	7326	7461	7326
q19	1675	1592	1575	1575
q20	2046	1807	1823	1807
q21	5404	5287	5236	5236
q22	1129	1082	1037	1037
Total cold run time: 63495 ms
Total hot run time: 54667 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186348 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b64e00b02a5dc3c860e6bf1db6d1c999cf249b8, data reload: false

query1	897	364	358	358
query2	6457	1955	1902	1902
query3	6647	212	220	212
query4	34364	23079	23042	23042
query5	4155	505	484	484
query6	262	160	170	160
query7	4587	296	289	289
query8	266	207	212	207
query9	8446	2458	2447	2447
query10	429	279	273	273
query11	16492	15034	15269	15034
query12	150	106	98	98
query13	1644	395	381	381
query14	9779	6842	7106	6842
query15	252	166	175	166
query16	7885	436	455	436
query17	1566	554	553	553
query18	2009	284	287	284
query19	187	148	145	145
query20	120	115	112	112
query21	230	101	99	99
query22	4397	4160	4104	4104
query23	34073	33378	33375	33375
query24	11106	2828	2861	2828
query25	636	381	403	381
query26	1225	156	158	156
query27	2432	278	279	278
query28	6821	2034	2011	2011
query29	802	400	404	400
query30	303	162	147	147
query31	1014	752	775	752
query32	95	55	61	55
query33	756	296	289	289
query34	965	478	484	478
query35	880	716	733	716
query36	1107	902	919	902
query37	164	82	81	81
query38	3982	3827	3900	3827
query39	1445	1372	1392	1372
query40	199	117	118	117
query41	48	48	45	45
query42	114	94	98	94
query43	513	476	459	459
query44	1226	740	768	740
query45	196	168	167	167
query46	1104	743	767	743
query47	1873	1764	1767	1764
query48	360	293	296	293
query49	1103	428	417	417
query50	801	416	423	416
query51	7168	6947	7052	6947
query52	97	90	84	84
query53	257	189	177	177
query54	954	457	466	457
query55	77	78	74	74
query56	278	273	262	262
query57	1170	1081	1070	1070
query58	231	214	230	214
query59	2969	2922	2756	2756
query60	290	263	269	263
query61	134	98	102	98
query62	834	639	658	639
query63	214	189	185	185
query64	5380	2249	1744	1744
query65	3222	3189	3157	3157
query66	1374	338	335	335
query67	15437	15268	15265	15265
query68	3207	581	631	581
query69	406	278	277	277
query70	1179	1025	1072	1025
query71	332	272	272	272
query72	6263	2305	2117	2117
query73	759	326	334	326
query74	9052	8763	8768	8763
query75	3375	2660	2776	2660
query76	1844	929	1008	929
query77	464	321	308	308
query78	10723	9222	9620	9222
query79	1077	529	525	525
query80	739	496	490	490
query81	492	236	239	236
query82	234	137	139	137
query83	178	158	153	153
query84	224	81	78	78
query85	689	371	274	274
query86	305	291	300	291
query87	4362	4270	4184	4184
query88	2886	2364	2333	2333
query89	381	289	288	288
query90	1935	200	198	198
query91	126	96	100	96
query92	68	56	52	52
query93	1024	556	537	537
query94	882	303	283	283
query95	362	263	263	263
query96	590	266	269	266
query97	3168	3083	3062	3062
query98	217	211	204	204
query99	1469	1260	1273	1260
Total cold run time: 285662 ms
Total hot run time: 186348 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7b64e00b02a5dc3c860e6bf1db6d1c999cf249b8, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.05
query3	0.22	0.05	0.05
query4	1.66	0.10	0.10
query5	0.50	0.51	0.50
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.49	0.50
query10	0.54	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.11	0.11
query13	0.61	0.59	0.58
query14	0.78	0.78	0.78
query15	0.88	0.82	0.82
query16	0.34	0.37	0.38
query17	1.05	1.04	1.08
query18	0.21	0.20	0.20
query19	1.95	1.83	1.79
query20	0.01	0.01	0.01
query21	15.42	0.67	0.66
query22	4.75	6.59	2.59
query23	18.32	1.34	1.31
query24	2.11	0.23	0.22
query25	0.14	0.08	0.08
query26	0.26	0.17	0.18
query27	0.08	0.08	0.07
query28	13.27	1.00	1.00
query29	12.64	3.39	3.31
query30	0.24	0.06	0.06
query31	2.88	0.40	0.41
query32	3.24	0.48	0.46
query33	2.93	3.01	3.00
query34	16.78	4.39	4.42
query35	4.42	4.44	4.45
query36	0.65	0.50	0.48
query37	0.18	0.16	0.15
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.06
query42	0.06	0.06	0.06
query43	0.05	0.05	0.04
Total cold run time: 109.82 s
Total hot run time: 31.61 s

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38317 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf353c4b8db82dd13b1be886926fa23e7c54642c, data reload: false

------ Round 1 ----------------------------------
q1	17621	4444	4266	4266
q2	2030	184	177	177
q3	10455	1209	1105	1105
q4	10138	779	735	735
q5	7718	2825	2837	2825
q6	227	139	140	139
q7	967	626	604	604
q8	9330	2074	2031	2031
q9	7320	6554	6561	6554
q10	7004	2208	2218	2208
q11	458	252	256	252
q12	405	224	223	223
q13	18032	3075	3047	3047
q14	285	246	240	240
q15	517	489	524	489
q16	482	393	400	393
q17	997	687	676	676
q18	7518	7025	7001	7001
q19	1382	1148	984	984
q20	699	332	322	322
q21	3946	3026	3167	3026
q22	1135	1025	1020	1020
Total cold run time: 108666 ms
Total hot run time: 38317 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4364	4305	4327	4305
q2	380	289	269	269
q3	2961	2682	2711	2682
q4	1946	1686	1676	1676
q5	5472	5467	5452	5452
q6	220	130	135	130
q7	2157	1743	1785	1743
q8	3182	3388	3350	3350
q9	8485	8465	8455	8455
q10	3525	3278	3228	3228
q11	605	494	501	494
q12	803	616	639	616
q13	16776	3046	3085	3046
q14	304	302	273	273
q15	530	485	477	477
q16	515	426	424	424
q17	1820	1503	1495	1495
q18	7842	7593	7461	7461
q19	1663	1553	1480	1480
q20	2064	1832	1873	1832
q21	5469	5240	5241	5240
q22	1127	1066	1037	1037
Total cold run time: 72210 ms
Total hot run time: 55165 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187207 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf353c4b8db82dd13b1be886926fa23e7c54642c, data reload: false

query1	907	376	365	365
query2	6470	1944	1860	1860
query3	6656	205	215	205
query4	33858	23252	23219	23219
query5	4206	506	493	493
query6	262	163	166	163
query7	4592	303	296	296
query8	267	210	216	210
query9	8446	2448	2440	2440
query10	449	272	271	271
query11	17842	15080	15022	15022
query12	155	101	98	98
query13	1641	380	380	380
query14	9993	7304	7409	7304
query15	258	167	167	167
query16	7833	456	459	456
query17	1702	560	550	550
query18	1402	292	292	292
query19	309	147	145	145
query20	124	107	107	107
query21	206	103	99	99
query22	4462	4098	4082	4082
query23	34167	33739	33465	33465
query24	11067	2784	2845	2784
query25	611	406	417	406
query26	1099	154	157	154
query27	2303	280	283	280
query28	7328	2030	2010	2010
query29	707	413	400	400
query30	286	161	148	148
query31	980	777	773	773
query32	106	57	59	57
query33	749	296	285	285
query34	977	477	479	477
query35	879	711	715	711
query36	1055	930	923	923
query37	146	86	84	84
query38	3978	3852	3856	3852
query39	1442	1423	1388	1388
query40	197	146	115	115
query41	48	47	44	44
query42	118	97	97	97
query43	498	453	462	453
query44	1225	742	768	742
query45	202	164	168	164
query46	1094	738	741	738
query47	1867	1790	1795	1790
query48	373	296	300	296
query49	1096	432	417	417
query50	818	408	413	408
query51	7102	7118	7120	7118
query52	97	86	91	86
query53	252	180	177	177
query54	928	466	460	460
query55	79	78	81	78
query56	277	247	259	247
query57	1204	1080	1058	1058
query58	238	217	262	217
query59	2928	2859	2710	2710
query60	337	261	264	261
query61	103	100	104	100
query62	833	654	661	654
query63	220	186	181	181
query64	5287	2283	1768	1768
query65	3249	3186	3156	3156
query66	1273	357	346	346
query67	15602	15362	15282	15282
query68	3571	594	578	578
query69	403	279	305	279
query70	1144	1038	1125	1038
query71	345	286	328	286
query72	6230	2286	2112	2112
query73	743	325	323	323
query74	9180	8824	8958	8824
query75	3398	2696	2691	2691
query76	1910	1017	925	925
query77	469	315	314	314
query78	9759	9484	9008	9008
query79	1034	544	519	519
query80	701	501	504	501
query81	521	235	230	230
query82	233	135	144	135
query83	177	151	151	151
query84	224	78	77	77
query85	670	279	284	279
query86	332	299	278	278
query87	4389	4261	4194	4194
query88	2942	2449	2353	2353
query89	372	289	284	284
query90	1819	200	197	197
query91	121	96	101	96
query92	60	50	51	50
query93	1025	540	540	540
query94	788	307	308	307
query95	358	262	268	262
query96	578	271	272	271
query97	3173	3077	3148	3077
query98	218	203	197	197
query99	1478	1276	1247	1247
Total cold run time: 285830 ms
Total hot run time: 187207 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf353c4b8db82dd13b1be886926fa23e7c54642c, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.08
query5	0.51	0.49	0.50
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.56	0.50	0.50
query10	0.54	0.53	0.53
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.58	0.58
query14	0.78	0.80	0.77
query15	0.85	0.83	0.83
query16	0.37	0.36	0.38
query17	1.04	1.05	1.03
query18	0.20	0.19	0.21
query19	1.92	1.80	1.85
query20	0.02	0.01	0.01
query21	15.40	0.67	0.68
query22	4.41	7.37	1.67
query23	18.29	1.38	1.30
query24	2.17	0.23	0.22
query25	0.16	0.09	0.07
query26	0.27	0.18	0.18
query27	0.07	0.07	0.08
query28	13.21	1.00	1.00
query29	12.63	3.35	3.34
query30	0.25	0.06	0.05
query31	2.86	0.39	0.40
query32	3.26	0.48	0.48
query33	2.97	2.98	2.98
query34	17.17	4.40	4.37
query35	4.46	4.40	4.46
query36	0.65	0.47	0.50
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.15	0.12	0.13
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.88 s
Total hot run time: 30.55 s

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6df6ee0456f7e5212c1f6316632d92645c1e0895, data reload: false

------ Round 1 ----------------------------------
q1	17606	4327	4342	4327
q2	2022	184	191	184
q3	11771	985	1097	985
q4	10510	815	771	771
q5	7770	2799	2816	2799
q6	222	137	136	136
q7	989	602	608	602
q8	9346	2087	2071	2071
q9	7245	6505	6554	6505
q10	7001	2273	2232	2232
q11	470	241	239	239
q12	392	221	227	221
q13	18821	2993	3010	2993
q14	275	227	243	227
q15	525	484	472	472
q16	483	401	394	394
q17	974	635	744	635
q18	7215	6886	6946	6886
q19	1388	1048	1063	1048
q20	688	330	320	320
q21	3933	3021	2921	2921
q22	1098	998	997	997
Total cold run time: 110744 ms
Total hot run time: 37965 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4362	4325	4266	4266
q2	374	277	262	262
q3	2879	2646	2694	2646
q4	1908	1653	1655	1653
q5	5594	5678	5781	5678
q6	220	134	133	133
q7	2234	1831	1833	1831
q8	3308	3504	3500	3500
q9	8845	8805	8792	8792
q10	3591	3416	3387	3387
q11	608	503	509	503
q12	822	700	655	655
q13	14458	3184	3223	3184
q14	323	292	287	287
q15	552	496	495	495
q16	499	451	447	447
q17	1847	1525	1555	1525
q18	8047	7710	7824	7710
q19	1725	1510	1665	1510
q20	2170	1902	1947	1902
q21	5736	5382	5439	5382
q22	1186	1060	1041	1041
Total cold run time: 71288 ms
Total hot run time: 56789 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192570 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6df6ee0456f7e5212c1f6316632d92645c1e0895, data reload: false

query1	1256	905	860	860
query2	6288	1920	1908	1908
query3	10608	3914	4056	3914
query4	59429	27306	23208	23208
query5	5389	503	505	503
query6	401	162	173	162
query7	5782	293	291	291
query8	300	216	215	215
query9	9136	2440	2438	2438
query10	509	266	264	264
query11	17843	15171	15340	15171
query12	163	116	103	103
query13	1587	400	405	400
query14	10313	7172	7271	7172
query15	224	181	169	169
query16	7580	476	475	475
query17	1102	588	608	588
query18	2125	310	295	295
query19	287	144	144	144
query20	124	109	107	107
query21	198	113	105	105
query22	4591	4438	4432	4432
query23	34201	33580	33564	33564
query24	5951	2847	2847	2847
query25	518	371	376	371
query26	680	153	152	152
query27	1753	290	276	276
query28	3661	2024	2008	2008
query29	677	396	407	396
query30	231	152	157	152
query31	911	767	745	745
query32	83	51	53	51
query33	428	275	277	275
query34	854	476	466	466
query35	840	729	694	694
query36	1052	899	931	899
query37	136	82	83	82
query38	4013	3868	3885	3868
query39	1445	1371	1399	1371
query40	195	114	114	114
query41	45	45	44	44
query42	117	97	98	97
query43	503	468	468	468
query44	1142	748	746	746
query45	190	168	165	165
query46	1091	723	749	723
query47	1907	1778	1782	1778
query48	382	290	290	290
query49	749	417	430	417
query50	827	416	410	410
query51	7241	7098	7110	7098
query52	98	89	88	88
query53	252	179	183	179
query54	566	464	455	455
query55	80	79	80	79
query56	294	264	271	264
query57	1169	1089	1059	1059
query58	220	246	296	246
query59	2928	2833	2932	2833
query60	295	266	269	266
query61	103	99	99	99
query62	726	629	642	629
query63	214	183	181	181
query64	3162	1764	1781	1764
query65	3213	3178	3146	3146
query66	662	327	344	327
query67	15303	15432	15162	15162
query68	2989	573	578	573
query69	397	275	277	275
query70	1135	1098	1112	1098
query71	351	278	279	278
query72	2605	2103	2081	2081
query73	715	326	330	326
query74	9111	8859	8865	8859
query75	3400	2632	2705	2632
query76	1459	1018	957	957
query77	529	323	324	323
query78	9504	10634	9587	9587
query79	1027	556	533	533
query80	678	504	504	504
query81	529	231	227	227
query82	266	139	138	138
query83	171	146	207	146
query84	266	79	78	78
query85	673	284	302	284
query86	303	286	284	284
query87	4439	4272	4326	4272
query88	2885	2381	2371	2371
query89	375	290	284	284
query90	1928	199	194	194
query91	118	99	102	99
query92	57	54	51	51
query93	1021	540	533	533
query94	694	289	302	289
query95	328	262	257	257
query96	587	271	274	271
query97	3212	3083	3065	3065
query98	215	208	198	198
query99	1590	1269	1235	1235
Total cold run time: 299900 ms
Total hot run time: 192570 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6df6ee0456f7e5212c1f6316632d92645c1e0895, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.08
query5	0.50	0.51	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.49	0.49
query10	0.53	0.54	0.54
query11	0.15	0.12	0.11
query12	0.16	0.12	0.13
query13	0.62	0.59	0.60
query14	0.77	0.80	0.77
query15	0.84	0.83	0.82
query16	0.37	0.37	0.37
query17	1.04	1.05	1.10
query18	0.21	0.20	0.20
query19	1.84	1.69	1.88
query20	0.01	0.01	0.01
query21	15.39	0.67	0.66
query22	4.06	7.88	1.70
query23	18.22	1.36	1.21
query24	2.04	0.24	0.22
query25	0.16	0.09	0.09
query26	0.27	0.18	0.17
query27	0.08	0.07	0.07
query28	13.28	1.02	0.99
query29	12.60	3.34	3.28
query30	0.24	0.05	0.06
query31	2.87	0.40	0.39
query32	3.26	0.47	0.47
query33	2.99	2.97	3.01
query34	16.98	4.38	4.35
query35	4.40	4.46	4.40
query36	0.64	0.47	0.49
query37	0.20	0.15	0.17
query38	0.16	0.15	0.16
query39	0.04	0.04	0.04
query40	0.15	0.13	0.14
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 109.03 s
Total hot run time: 30.33 s

@Jibing-Li
Copy link
Contributor Author

run cloud_p1

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38191 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 68529568e3b948d4a6c80d1dc7ce57b3c6bc7b50, data reload: false

------ Round 1 ----------------------------------
q1	17638	4536	4363	4363
q2	2019	189	182	182
q3	11685	967	1012	967
q4	10518	725	732	725
q5	7770	2836	2787	2787
q6	227	140	140	140
q7	991	615	596	596
q8	9335	2089	2063	2063
q9	6987	6534	6550	6534
q10	6994	2260	2209	2209
q11	465	247	248	247
q12	394	223	231	223
q13	18869	3053	3014	3014
q14	285	229	239	229
q15	534	485	488	485
q16	594	500	508	500
q17	980	669	681	669
q18	7280	6850	6975	6850
q19	1389	1085	1021	1021
q20	701	351	332	332
q21	4017	3032	3076	3032
q22	1135	1023	1023	1023
Total cold run time: 110807 ms
Total hot run time: 38191 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4392	4407	4278	4278
q2	376	282	278	278
q3	2886	2657	2657	2657
q4	1928	1630	1672	1630
q5	5654	5686	5756	5686
q6	226	142	147	142
q7	2256	1863	1840	1840
q8	3312	3432	3514	3432
q9	8750	8872	8788	8788
q10	3639	3378	3315	3315
q11	620	522	548	522
q12	823	690	648	648
q13	14257	3206	3261	3206
q14	332	308	291	291
q15	537	507	493	493
q16	627	607	597	597
q17	1825	1569	1529	1529
q18	8171	7784	7920	7784
q19	1722	1586	1583	1583
q20	2116	1923	1916	1916
q21	5873	5455	5574	5455
q22	1105	1062	1023	1023
Total cold run time: 71427 ms
Total hot run time: 57093 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193016 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 68529568e3b948d4a6c80d1dc7ce57b3c6bc7b50, data reload: false

query1	1256	894	859	859
query2	6367	2044	1962	1962
query3	10598	3836	3908	3836
query4	59680	27093	23363	23363
query5	5520	542	528	528
query6	417	169	160	160
query7	5772	296	295	295
query8	293	221	217	217
query9	9033	2517	2506	2506
query10	502	273	264	264
query11	16089	15137	15374	15137
query12	165	109	106	106
query13	1559	425	406	406
query14	11217	7151	7454	7151
query15	250	175	181	175
query16	7599	475	524	475
query17	1077	573	570	570
query18	2055	297	302	297
query19	285	144	143	143
query20	118	109	106	106
query21	205	102	102	102
query22	4479	4468	4429	4429
query23	34374	33538	33525	33525
query24	5984	2873	2829	2829
query25	535	396	387	387
query26	698	156	154	154
query27	1789	282	279	279
query28	4025	2141	2109	2109
query29	703	401	407	401
query30	237	157	159	157
query31	929	770	774	770
query32	84	59	55	55
query33	414	285	279	279
query34	855	479	472	472
query35	846	718	711	711
query36	1063	911	926	911
query37	143	91	90	90
query38	4079	3910	3849	3849
query39	1437	1417	1382	1382
query40	192	122	114	114
query41	49	48	44	44
query42	116	96	100	96
query43	512	476	493	476
query44	1114	743	752	743
query45	199	171	174	171
query46	1076	731	766	731
query47	1859	1801	1820	1801
query48	382	298	301	298
query49	758	418	439	418
query50	835	421	429	421
query51	7235	7062	6932	6932
query52	101	89	95	89
query53	251	183	185	183
query54	563	461	474	461
query55	79	78	89	78
query56	298	265	287	265
query57	1201	1062	1077	1062
query58	226	320	229	229
query59	3037	2791	2859	2791
query60	292	268	272	268
query61	99	100	96	96
query62	759	657	655	655
query63	217	187	189	187
query64	2846	691	703	691
query65	3240	3132	3217	3132
query66	607	337	343	337
query67	15484	15208	15042	15042
query68	2965	581	598	581
query69	408	280	291	280
query70	1182	1146	1100	1100
query71	342	283	273	273
query72	5771	4035	3998	3998
query73	759	338	336	336
query74	9258	8862	8885	8862
query75	3371	2661	2757	2661
query76	1437	1029	998	998
query77	531	327	333	327
query78	10696	9367	9042	9042
query79	1044	539	525	525
query80	695	514	512	512
query81	466	245	239	239
query82	242	155	154	154
query83	169	148	149	148
query84	253	79	81	79
query85	736	300	285	285
query86	326	305	288	288
query87	4435	4292	4238	4238
query88	3506	2328	2353	2328
query89	378	281	289	281
query90	1988	200	197	197
query91	122	99	99	99
query92	62	51	54	51
query93	1054	530	534	530
query94	650	308	295	295
query95	327	263	265	263
query96	595	267	273	267
query97	3197	3135	3051	3051
query98	217	201	197	197
query99	1536	1281	1273	1273
Total cold run time: 305076 ms
Total hot run time: 193016 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 68529568e3b948d4a6c80d1dc7ce57b3c6bc7b50, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.06	0.04
query4	1.67	0.08	0.08
query5	0.51	0.51	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.50
query10	0.54	0.55	0.54
query11	0.15	0.12	0.12
query12	0.15	0.13	0.12
query13	0.62	0.59	0.59
query14	2.07	2.05	2.06
query15	0.85	0.81	0.82
query16	0.37	0.37	0.37
query17	0.96	0.98	1.04
query18	0.21	0.21	0.20
query19	1.91	1.77	1.78
query20	0.02	0.01	0.01
query21	15.39	0.68	0.65
query22	4.35	6.16	2.60
query23	18.29	1.42	1.27
query24	2.04	0.25	0.22
query25	0.14	0.08	0.07
query26	0.28	0.18	0.18
query27	0.07	0.07	0.07
query28	13.25	1.02	1.01
query29	12.60	3.38	3.35
query30	0.24	0.05	0.06
query31	2.87	0.42	0.40
query32	3.24	0.50	0.47
query33	2.92	2.99	3.01
query34	16.97	4.46	4.36
query35	4.41	4.45	4.47
query36	0.65	0.50	0.49
query37	0.18	0.17	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.57 s
Total hot run time: 32.76 s

@Jibing-Li
Copy link
Contributor Author

run external

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

morningman
morningman approved these changes Aug 28, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Jibing-Li Jibing-Li merged commit 06e095c into apache:master Aug 28, 2024
28 of 30 checks passed
@Jibing-Li Jibing-Li deleted the cleaner branch August 28, 2024 14:23
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
380858d 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
e5bde04 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
This was referenced Sep 2, 2024
Merged
Merged
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
8e18701 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
3d9bff5 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
00de5d9 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
87adf64 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
92f439e 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
de501a6 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
c546988 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (#39779) (#40252)
1e925d9 
backport: #39779
Jibing-Li added a commit that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (#39779) (#40253)
a5efdc6 
backport: #39779
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@Jibing-Li @dataroaring
[improve](statistics)Clean expired TableStatsMeta. (#39779)
386cb86 
Drop catalog and older version of drop database (before
#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Jibing-Li @doris-robot @morningman @morrySnow