Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](statistics)Drop table stats in unprotectDropTable so that dropDB could call it as well. #39685

Merged
merged 1 commit into from
Aug 22, 2024
Merged

[improvement](statistics)Drop table stats in unprotectDropTable so that dropDB could call it as well. #39685

merged 1 commit into from
Aug 22, 2024

Conversation

Jibing-Li
Copy link
Contributor

Drop table stats in unprotectDropTable so that dropDB could call it as well.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li changed the title [fix](statistics)Drop table stats in unprotectDropTable so that dropDB could call it as well. [improvement](statistics)Drop table stats in unprotectDropTable so that dropDB could call it as well. Aug 21, 2024
@Jibing-Li Jibing-Li marked this pull request as ready for review August 21, 2024 04:09
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38282 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a63c186c5a31cac496a8cfea33e370dbfab06ec, data reload: false

------ Round 1 ----------------------------------
q1	17866	4346	4330	4330
q2	2045	223	233	223
q3	10394	1173	1110	1110
q4	10172	715	720	715
q5	7776	2904	2825	2825
q6	270	161	165	161
q7	1023	674	668	668
q8	9475	2072	2095	2072
q9	7388	6551	6526	6526
q10	7076	2231	2228	2228
q11	495	278	277	277
q12	444	267	264	264
q13	17961	3017	3009	3009
q14	309	266	257	257
q15	564	530	530	530
q16	543	423	420	420
q17	987	670	752	670
q18	7509	6896	6811	6811
q19	7612	1097	1041	1041
q20	732	361	363	361
q21	4092	2750	2875	2750
q22	1147	1034	1045	1034
Total cold run time: 115880 ms
Total hot run time: 38282 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4505	4268	4267	4267
q2	420	325	313	313
q3	2861	2610	2644	2610
q4	1934	1644	1645	1644
q5	5365	5424	5368	5368
q6	243	153	158	153
q7	2109	1764	1710	1710
q8	3202	3358	3315	3315
q9	8410	8341	8436	8341
q10	3427	3137	3197	3137
q11	628	555	538	538
q12	804	617	613	613
q13	17536	2994	3031	2994
q14	321	295	282	282
q15	588	515	543	515
q16	493	448	443	443
q17	1806	1477	1500	1477
q18	7864	7538	7384	7384
q19	2848	1605	1611	1605
q20	2064	1815	1787	1787
q21	13843	5079	4980	4980
q22	1127	1040	1046	1040
Total cold run time: 82398 ms
Total hot run time: 54516 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191273 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a63c186c5a31cac496a8cfea33e370dbfab06ec, data reload: false

query1	979	402	392	392
query2	6752	2061	1966	1966
query3	6664	225	236	225
query4	29302	23171	23117	23117
query5	4377	734	708	708
query6	311	218	212	212
query7	4610	344	327	327
query8	485	439	441	439
query9	8663	2535	2506	2506
query10	502	353	338	338
query11	17856	15124	14993	14993
query12	195	142	138	138
query13	1708	443	449	443
query14	10444	7387	6864	6864
query15	296	205	204	204
query16	7478	522	545	522
query17	1668	627	586	586
query18	2034	370	342	342
query19	358	176	173	173
query20	149	142	139	139
query21	255	145	143	143
query22	4373	3957	4064	3957
query23	34014	33288	33565	33288
query24	11151	2885	2942	2885
query25	637	409	425	409
query26	906	182	178	178
query27	2249	310	308	308
query28	7172	2131	2101	2101
query29	776	464	443	443
query30	344	191	183	183
query31	1041	849	827	827
query32	123	81	85	81
query33	824	358	347	347
query34	916	529	514	514
query35	912	799	794	794
query36	1086	988	970	970
query37	172	108	105	105
query38	3938	3900	3912	3900
query39	1530	1457	1488	1457
query40	240	158	157	157
query41	145	141	142	141
query42	133	118	118	118
query43	536	500	512	500
query44	1262	787	812	787
query45	224	199	198	198
query46	1139	766	778	766
query47	1915	1823	1847	1823
query48	423	347	336	336
query49	1191	594	598	594
query50	860	465	475	465
query51	7174	7163	7090	7090
query52	122	113	113	113
query53	303	230	225	225
query54	1021	508	504	504
query55	90	89	91	89
query56	341	325	330	325
query57	1172	1117	1099	1099
query58	331	307	314	307
query59	3047	2770	2744	2744
query60	354	341	324	324
query61	152	158	151	151
query62	881	699	748	699
query63	263	232	226	226
query64	5531	2366	1860	1860
query65	3263	3160	3165	3160
query66	1192	685	684	684
query67	15674	14982	15025	14982
query68	6412	596	593	593
query69	723	426	332	332
query70	1228	1138	1208	1138
query71	533	327	319	319
query72	7491	2339	2098	2098
query73	829	363	363	363
query74	9208	8815	9003	8815
query75	4899	2764	2733	2733
query76	4278	1054	1024	1024
query77	892	452	468	452
query78	10797	9226	9888	9226
query79	8620	546	559	546
query80	1174	653	645	645
query81	631	264	269	264
query82	1210	165	171	165
query83	426	217	274	217
query84	297	99	102	99
query85	1568	356	358	356
query86	462	331	334	331
query87	4466	4312	4236	4236
query88	4383	2529	2529	2529
query89	455	334	335	334
query90	2060	239	238	238
query91	157	131	131	131
query92	96	77	78	77
query93	4937	560	555	555
query94	1094	323	338	323
query95	392	298	300	298
query96	623	285	286	285
query97	3223	3025	3093	3025
query98	251	228	231	228
query99	1551	1308	1307	1307
Total cold run time: 311595 ms
Total hot run time: 191273 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a63c186c5a31cac496a8cfea33e370dbfab06ec, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.05	0.05
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.54	0.49	0.48
query10	0.55	0.56	0.55
query11	0.16	0.12	0.12
query12	0.16	0.12	0.13
query13	0.60	0.59	0.58
query14	0.77	0.79	0.80
query15	0.88	0.82	0.82
query16	0.37	0.37	0.38
query17	0.95	1.06	1.04
query18	0.22	0.20	0.21
query19	1.94	1.84	1.83
query20	0.01	0.01	0.02
query21	15.40	0.67	0.66
query22	3.69	6.78	2.65
query23	18.29	1.42	1.26
query24	2.13	0.23	0.22
query25	0.15	0.09	0.08
query26	0.28	0.18	0.18
query27	0.09	0.08	0.08
query28	13.28	1.05	1.02
query29	12.69	3.31	3.36
query30	0.39	0.21	0.19
query31	2.81	0.41	0.39
query32	3.24	0.49	0.50
query33	2.96	2.96	2.96
query34	17.09	4.37	4.35
query35	4.47	4.39	4.47
query36	0.68	0.47	0.48
query37	0.21	0.18	0.19
query38	0.19	0.17	0.17
query39	0.07	0.06	0.06
query40	0.19	0.15	0.16
query41	0.12	0.08	0.07
query42	0.08	0.07	0.07
query43	0.08	0.06	0.06
Total cold run time: 109.48 s
Total hot run time: 31.87 s

@Jibing-Li
Copy link
Contributor Author

run p0

1 similar comment
@Jibing-Li
Copy link
Contributor Author

run p0

@Jibing-Li
Copy link
Contributor Author

run external

1 similar comment
@Jibing-Li
Copy link
Contributor Author

run external

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38695 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5979041c3f359348f3c11630cbc5cf0b9bfd663e, data reload: false

------ Round 1 ----------------------------------
q1	18209	4662	4410	4410
q2	2865	225	216	216
q3	10724	1208	1131	1131
q4	11050	773	845	773
q5	7890	2885	2900	2885
q6	266	159	166	159
q7	1030	683	651	651
q8	9421	2122	2076	2076
q9	7098	6574	6535	6535
q10	7094	2292	2250	2250
q11	499	278	278	278
q12	436	267	272	267
q13	17774	3010	2961	2961
q14	300	275	246	246
q15	551	519	524	519
q16	520	404	407	404
q17	1011	740	727	727
q18	7385	6898	6772	6772
q19	6403	1110	1088	1088
q20	703	343	360	343
q21	3840	2958	2954	2954
q22	1120	1052	1050	1050
Total cold run time: 116189 ms
Total hot run time: 38695 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5018	4331	4378	4331
q2	402	314	299	299
q3	2884	2597	2647	2597
q4	1900	1649	1654	1649
q5	5367	5403	5393	5393
q6	239	146	145	145
q7	2093	1705	1694	1694
q8	3215	3416	3372	3372
q9	8461	8413	8357	8357
q10	3408	3152	3152	3152
q11	625	567	539	539
q12	822	667	649	649
q13	16553	3023	3023	3023
q14	334	297	300	297
q15	568	527	515	515
q16	508	448	466	448
q17	1833	1498	1500	1498
q18	7681	7498	7401	7401
q19	2633	1504	1633	1504
q20	2099	1844	1838	1838
q21	12512	4993	5236	4993
q22	1158	1104	1068	1068
Total cold run time: 80313 ms
Total hot run time: 54762 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191408 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5979041c3f359348f3c11630cbc5cf0b9bfd663e, data reload: false

query1	946	421	389	389
query2	6746	2017	1911	1911
query3	6672	231	241	231
query4	34219	23416	23238	23238
query5	4465	723	701	701
query6	318	210	205	205
query7	4610	340	341	340
query8	484	428	427	427
query9	8489	2556	2543	2543
query10	528	356	348	348
query11	16666	14985	15150	14985
query12	189	138	137	137
query13	1724	458	459	458
query14	10314	6827	7156	6827
query15	261	191	191	191
query16	8055	535	498	498
query17	1595	596	588	588
query18	2153	328	339	328
query19	265	170	163	163
query20	147	137	136	136
query21	248	139	140	139
query22	4200	3940	3974	3940
query23	34352	33355	33486	33355
query24	11209	2871	2873	2871
query25	649	417	418	417
query26	1180	184	180	180
query27	2429	299	300	299
query28	7236	2158	2124	2124
query29	858	454	441	441
query30	336	192	184	184
query31	1054	827	831	827
query32	125	81	82	81
query33	809	348	338	338
query34	919	502	507	502
query35	899	746	753	746
query36	1123	968	937	937
query37	176	103	102	102
query38	3978	3782	3819	3782
query39	1518	1455	1450	1450
query40	240	156	153	153
query41	142	138	150	138
query42	137	116	120	116
query43	545	518	520	518
query44	1249	776	791	776
query45	221	193	196	193
query46	1124	749	759	749
query47	1926	1836	1871	1836
query48	409	335	334	334
query49	1272	579	580	579
query50	858	465	478	465
query51	7219	7146	7158	7146
query52	124	111	107	107
query53	295	235	222	222
query54	951	524	499	499
query55	92	92	91	91
query56	337	299	308	299
query57	1223	1114	1132	1114
query58	319	323	306	306
query59	3037	2852	2802	2802
query60	349	321	323	321
query61	148	149	150	149
query62	892	663	694	663
query63	253	222	236	222
query64	5349	2329	3396	2329
query65	3206	3194	3151	3151
query66	1158	707	672	672
query67	15522	15197	15215	15197
query68	5223	581	580	580
query69	745	396	312	312
query70	1209	1176	1152	1152
query71	531	309	309	309
query72	7638	2288	2112	2112
query73	832	358	354	354
query74	9351	8761	8858	8761
query75	4595	2711	2727	2711
query76	3944	1020	984	984
query77	935	436	463	436
query78	9752	9167	9702	9167
query79	7030	571	565	565
query80	2423	606	606	606
query81	636	265	258	258
query82	854	164	160	160
query83	375	217	213	213
query84	289	94	99	94
query85	1678	357	348	348
query86	489	303	329	303
query87	4451	4252	4283	4252
query88	4634	2442	2429	2429
query89	469	325	319	319
query90	2034	236	237	236
query91	157	128	136	128
query92	89	74	77	74
query93	4307	562	545	545
query94	1072	333	338	333
query95	400	301	297	297
query96	616	284	281	281
query97	3267	3111	3048	3048
query98	240	222	229	222
query99	1617	1331	1313	1313
Total cold run time: 312623 ms
Total hot run time: 191408 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5979041c3f359348f3c11630cbc5cf0b9bfd663e, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.07	0.08
query5	0.54	0.49	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.55	0.49	0.50
query10	0.54	0.54	0.54
query11	0.16	0.13	0.13
query12	0.16	0.13	0.14
query13	0.63	0.60	0.59
query14	0.78	0.78	0.79
query15	0.85	0.84	0.84
query16	0.38	0.39	0.37
query17	1.00	0.98	1.01
query18	0.23	0.21	0.21
query19	1.92	1.70	1.68
query20	0.01	0.02	0.01
query21	15.42	0.66	0.66
query22	3.81	7.68	1.97
query23	18.22	1.36	1.29
query24	2.14	0.22	0.22
query25	0.17	0.09	0.08
query26	0.26	0.19	0.18
query27	0.08	0.08	0.08
query28	13.22	1.03	1.02
query29	12.64	3.30	3.29
query30	0.44	0.25	0.24
query31	2.81	0.41	0.41
query32	3.24	0.50	0.49
query33	2.97	3.02	3.03
query34	17.02	4.39	4.38
query35	4.47	4.48	4.46
query36	0.68	0.50	0.48
query37	0.20	0.18	0.17
query38	0.18	0.17	0.18
query39	0.06	0.05	0.06
query40	0.19	0.15	0.15
query41	0.12	0.07	0.07
query42	0.08	0.08	0.07
query43	0.06	0.06	0.07
Total cold run time: 109.45 s
Total hot run time: 31.22 s

@Jibing-Li
Copy link
Contributor Author

run cloud_p0

@Jibing-Li
Copy link
Contributor Author

run cloud_p1

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Jibing-Li Jibing-Li merged commit 3792cfc into apache:master Aug 22, 2024
27 of 30 checks passed
@Jibing-Li Jibing-Li deleted the tablestats branch August 22, 2024 02:21
@Jibing-Li Jibing-Li mentioned this pull request Aug 22, 2024
Merged
Jibing-Li added a commit that referenced this pull request Aug 28, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (#39779)
06e095c 
Drop catalog and older version of drop database (before
#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
380858d 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
e5bde04 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
8e18701 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
3d9bff5 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
00de5d9 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
87adf64 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
92f439e 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
de501a6 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Sep 2, 2024
@Jibing-Li
[improve](statistics)Clean expired TableStatsMeta. (apache#39779)
c546988 
Drop catalog and older version of drop database (before
apache#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@Jibing-Li @dataroaring
[improvement](statistics)Drop table stats in unprotectDropTable so th…
7748255 
…at dropDB could call it as well. (#39685)

Drop table stats in unprotectDropTable so that dropDB could call it as
well.
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@Jibing-Li @dataroaring
[improve](statistics)Clean expired TableStatsMeta. (#39779)
386cb86 
Drop catalog and older version of drop database (before
#39685), will not remove table stats
object in memory. Need a background thread to clean the garbage.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Oct 14, 2024
@Jibing-Li
[improvement](statistics)Drop table stats in unprotectDropTable so th…
f540b1c 
…at dropDB could call it as well. (apache#39685)

Drop table stats in unprotectDropTable so that dropDB could call it as
well.
@Jibing-Li Jibing-Li mentioned this pull request Oct 14, 2024
Merged
Jibing-Li added a commit that referenced this pull request Oct 14, 2024
@Jibing-Li
[improvement](statistics)Drop table stats in unprotectDropTable so th…
345cc91 
…at dropDB could call it as well. (#39685) (#41801)

backport: #39685
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Jibing-Li @doris-robot @morningman @morrySnow