Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](compaction) disable cumu delete rowset from performing ordered data compaction #38742

Merged
merged 2 commits into from
Aug 4, 2024

Conversation

luwei16
Copy link
Contributor

@luwei16 luwei16 commented Aug 1, 2024

problem

When enable_delete_when_cumu_compaction = true, multiple delete rowsets may be merged into an empty data rowset through ordered data compaction, causing the delete predicate to be lost.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@luwei16
Copy link
Contributor Author

luwei16 commented Aug 1, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 1, 2024
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 1, 2024
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@luwei16
Copy link
Contributor Author

luwei16 commented Aug 1, 2024

run ShellCheck

@doris-robot
Copy link

TPC-H: Total hot run time: 41720 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6c10142b18025abcac4708ae78087f77db33447, data reload: false

------ Round 1 ----------------------------------
q1	18145	4268	4122	4122
q2	2486	202	198	198
q3	11338	1317	1380	1317
q4	11237	860	890	860
q5	7972	3048	3003	3003
q6	223	138	138	138
q7	1039	619	623	619
q8	9430	1901	1942	1901
q9	8409	6591	6616	6591
q10	8685	3811	3807	3807
q11	422	252	243	243
q12	407	227	221	221
q13	17760	2925	2923	2923
q14	276	253	251	251
q15	525	485	499	485
q16	493	383	385	383
q17	949	934	940	934
q18	8008	7225	7272	7225
q19	1426	1212	1209	1209
q20	558	321	334	321
q21	5234	4693	4705	4693
q22	342	281	276	276
Total cold run time: 115364 ms
Total hot run time: 41720 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4047	4001	3995	3995
q2	328	228	220	220
q3	3004	2976	2981	2976
q4	1850	1840	1852	1840
q5	5262	5237	5217	5217
q6	215	132	127	127
q7	2064	1703	1668	1668
q8	3167	3265	3232	3232
q9	8297	8279	8247	8247
q10	3736	3831	3840	3831
q11	545	449	449	449
q12	728	590	568	568
q13	15245	2967	2969	2967
q14	283	255	250	250
q15	519	471	468	468
q16	447	386	394	386
q17	1720	1695	1709	1695
q18	7727	7381	7249	7249
q19	1651	1664	1660	1660
q20	1958	1764	1736	1736
q21	5365	5178	5194	5178
q22	489	443	461	443
Total cold run time: 68647 ms
Total hot run time: 54402 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 167921 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6c10142b18025abcac4708ae78087f77db33447, data reload: false

query1	910	384	366	366
query2	6472	1700	1661	1661
query3	6656	218	221	218
query4	20115	17523	17238	17238
query5	4316	524	512	512
query6	298	181	171	171
query7	4609	296	293	293
query8	266	201	190	190
query9	8507	2346	2336	2336
query10	440	281	291	281
query11	10530	10108	10211	10108
query12	129	88	87	87
query13	1614	368	366	366
query14	9454	6067	7490	6067
query15	193	160	170	160
query16	7109	445	463	445
query17	933	555	548	548
query18	1920	276	277	276
query19	188	141	139	139
query20	94	87	86	86
query21	202	100	105	100
query22	4386	4216	4070	4070
query23	33575	33380	33169	33169
query24	10457	3062	3026	3026
query25	672	405	427	405
query26	1757	145	148	145
query27	2940	275	275	275
query28	6931	1938	1932	1932
query29	1309	413	400	400
query30	292	152	147	147
query31	928	754	748	748
query32	101	56	56	56
query33	703	316	311	311
query34	883	480	480	480
query35	852	742	693	693
query36	968	859	867	859
query37	299	81	77	77
query38	2910	2785	2753	2753
query39	844	828	800	800
query40	284	112	110	110
query41	49	79	44	44
query42	131	101	99	99
query43	450	413	425	413
query44	1180	709	712	709
query45	201	175	175	175
query46	1076	810	814	810
query47	1808	1745	1711	1711
query48	366	290	284	284
query49	1192	412	430	412
query50	894	434	423	423
query51	6817	6702	6649	6649
query52	98	91	89	89
query53	248	176	178	176
query54	645	449	440	440
query55	77	73	73	73
query56	263	254	262	254
query57	1145	1041	1068	1041
query58	272	256	267	256
query59	2640	2512	2411	2411
query60	300	266	270	266
query61	96	93	110	93
query62	918	657	684	657
query63	207	179	178	178
query64	5817	1892	1894	1892
query65	3162	3094	3095	3094
query66	1466	335	330	330
query67	15191	14947	14903	14903
query68	4299	571	568	568
query69	449	296	299	296
query70	1107	1032	1081	1032
query71	358	273	275	273
query72	7059	2724	2475	2475
query73	769	330	328	328
query74	6098	5627	5686	5627
query75	3320	2731	2702	2702
query76	2295	1221	1267	1221
query77	447	307	302	302
query78	9452	8866	8874	8866
query79	1719	526	544	526
query80	1249	494	494	494
query81	555	224	225	224
query82	1112	132	132	132
query83	233	171	176	171
query84	275	76	77	76
query85	1251	313	298	298
query86	377	309	304	304
query87	3250	3140	3108	3108
query88	2865	2401	2391	2391
query89	384	284	282	282
query90	1767	186	191	186
query91	123	97	100	97
query92	64	49	49	49
query93	1476	598	606	598
query94	889	295	342	295
query95	372	263	262	262
query96	591	276	273	273
query97	3242	3099	3028	3028
query98	208	208	189	189
query99	1635	1292	1262	1262
Total cold run time: 262045 ms
Total hot run time: 167921 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6c10142b18025abcac4708ae78087f77db33447, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.03
query3	0.22	0.04	0.04
query4	1.68	0.06	0.07
query5	0.49	0.49	0.49
query6	1.15	0.71	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.52	0.50
query10	0.56	0.56	0.57
query11	0.15	0.12	0.11
query12	0.15	0.12	0.13
query13	0.62	0.59	0.60
query14	0.77	0.79	0.78
query15	0.89	0.87	0.86
query16	0.36	0.36	0.35
query17	0.98	1.00	1.02
query18	0.22	0.22	0.21
query19	1.88	1.70	1.73
query20	0.02	0.01	0.01
query21	15.41	0.77	0.67
query22	4.26	7.14	1.16
query23	17.77	1.42	1.34
query24	2.26	0.22	0.21
query25	0.19	0.08	0.08
query26	0.32	0.21	0.21
query27	0.45	0.24	0.22
query28	13.19	1.00	0.97
query29	12.56	3.29	3.26
query30	0.26	0.06	0.06
query31	2.87	0.41	0.40
query32	3.23	0.49	0.47
query33	2.94	2.97	2.97
query34	15.46	4.22	4.24
query35	4.28	4.30	4.29
query36	0.67	0.48	0.51
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.03	0.03
query40	0.15	0.14	0.14
query41	0.10	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.77 s
Total hot run time: 29.83 s

@luwei16
Copy link
Contributor Author

luwei16 commented Aug 2, 2024

run buildall

Copy link
Contributor

github-actions bot commented Aug 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41436 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35d19416ab7577648d943a1cec99cd7a69bb0f29, data reload: false

------ Round 1 ----------------------------------
q1	17623	4034	4073	4034
q2	2021	201	196	196
q3	10461	1291	1320	1291
q4	10170	757	918	757
q5	7645	2934	2950	2934
q6	219	140	141	140
q7	1056	609	612	609
q8	9431	1877	1937	1877
q9	8442	6633	6627	6627
q10	8747	3856	3852	3852
q11	437	247	242	242
q12	465	227	224	224
q13	17771	2938	2913	2913
q14	273	245	240	240
q15	519	487	498	487
q16	529	395	389	389
q17	969	940	927	927
q18	7987	7257	7235	7235
q19	1595	1222	1216	1216
q20	569	322	331	322
q21	5241	4694	4643	4643
q22	344	281	282	281
Total cold run time: 112514 ms
Total hot run time: 41436 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4131	4037	4086	4037
q2	325	221	218	218
q3	2999	2999	3095	2999
q4	1963	1973	1950	1950
q5	5551	5451	5433	5433
q6	223	135	133	133
q7	2159	1759	1839	1759
q8	3358	3354	3311	3311
q9	8634	8614	8714	8614
q10	3979	4093	3925	3925
q11	559	466	460	460
q12	760	601	649	601
q13	16413	3134	3099	3099
q14	293	276	267	267
q15	550	482	491	482
q16	475	411	418	411
q17	1796	1752	1718	1718
q18	8085	7752	7600	7600
q19	1722	1707	1723	1707
q20	2054	1882	1807	1807
q21	5623	5400	5253	5253
q22	533	457	465	457
Total cold run time: 72185 ms
Total hot run time: 56241 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170374 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35d19416ab7577648d943a1cec99cd7a69bb0f29, data reload: false

query1	925	381	372	372
query2	6472	1611	1667	1611
query3	6653	208	219	208
query4	19138	17307	17432	17307
query5	3664	517	519	517
query6	271	176	188	176
query7	4611	298	299	298
query8	255	200	200	200
query9	8530	2339	2335	2335
query10	438	294	261	261
query11	10397	10111	10157	10111
query12	116	87	91	87
query13	1631	374	375	374
query14	9937	7690	7256	7256
query15	209	156	160	156
query16	6748	433	493	433
query17	935	559	538	538
query18	1874	282	278	278
query19	188	144	141	141
query20	89	84	82	82
query21	197	102	101	101
query22	4180	3978	4012	3978
query23	33601	33720	33617	33617
query24	10283	3117	3151	3117
query25	690	408	423	408
query26	1779	154	150	150
query27	2758	291	285	285
query28	7368	2008	1962	1962
query29	1251	439	453	439
query30	244	153	155	153
query31	962	797	788	788
query32	99	60	61	60
query33	683	328	337	328
query34	984	515	504	504
query35	895	795	779	779
query36	1062	867	869	867
query37	282	89	86	86
query38	2952	2772	2839	2772
query39	886	837	817	817
query40	253	118	114	114
query41	49	45	46	45
query42	121	104	99	99
query43	462	431	417	417
query44	1196	746	755	746
query45	213	177	179	177
query46	1093	843	803	803
query47	1823	1719	1736	1719
query48	385	303	297	297
query49	989	423	432	423
query50	902	435	437	435
query51	6815	6711	6701	6701
query52	110	89	91	89
query53	260	188	177	177
query54	616	464	469	464
query55	77	74	79	74
query56	271	251	255	251
query57	1140	1037	1046	1037
query58	266	291	263	263
query59	2519	2473	2579	2473
query60	296	279	281	279
query61	97	94	92	92
query62	891	652	658	652
query63	214	185	181	181
query64	5706	1884	1867	1867
query65	3271	3091	3102	3091
query66	1313	339	323	323
query67	15204	15050	14890	14890
query68	4790	570	592	570
query69	670	430	315	315
query70	1101	1075	1046	1046
query71	457	281	278	278
query72	7612	2686	2483	2483
query73	883	330	331	330
query74	6059	5627	5660	5627
query75	3560	2752	2741	2741
query76	3369	1264	1290	1264
query77	592	310	319	310
query78	9464	8957	8946	8946
query79	1124	541	537	537
query80	939	497	501	497
query81	568	223	223	223
query82	451	128	139	128
query83	252	169	167	167
query84	265	78	77	77
query85	1198	300	285	285
query86	456	282	276	276
query87	3286	3145	3057	3057
query88	3003	2526	2515	2515
query89	381	296	286	286
query90	1670	195	192	192
query91	127	98	103	98
query92	59	48	49	48
query93	1389	618	623	618
query94	798	299	289	289
query95	445	278	274	274
query96	605	294	286	286
query97	3217	3005	3109	3005
query98	232	198	205	198
query99	1585	1283	1277	1277
Total cold run time: 261396 ms
Total hot run time: 170374 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35d19416ab7577648d943a1cec99cd7a69bb0f29, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.49	0.47
query6	1.15	0.73	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.50	0.51
query10	0.56	0.56	0.56
query11	0.16	0.12	0.11
query12	0.16	0.12	0.13
query13	0.62	0.60	0.60
query14	0.78	0.80	0.80
query15	0.90	0.85	0.87
query16	0.36	0.35	0.34
query17	1.01	1.00	0.96
query18	0.22	0.22	0.22
query19	1.81	1.74	1.74
query20	0.01	0.01	0.02
query21	15.39	0.76	0.65
query22	3.94	7.63	1.64
query23	18.11	1.39	1.30
query24	2.28	0.23	0.23
query25	0.18	0.08	0.08
query26	0.33	0.21	0.21
query27	0.47	0.25	0.23
query28	13.16	1.00	0.97
query29	12.53	3.51	3.41
query30	0.25	0.06	0.05
query31	2.86	0.40	0.39
query32	3.25	0.49	0.48
query33	2.95	3.02	2.96
query34	15.43	4.27	4.25
query35	4.31	4.27	4.27
query36	0.68	0.47	0.48
query37	0.19	0.16	0.17
query38	0.16	0.16	0.16
query39	0.04	0.04	0.03
query40	0.16	0.14	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 107.77 s
Total hot run time: 30.44 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit dfdec4d into apache:master Aug 4, 2024
27 of 28 checks passed
dataroaring pushed a commit that referenced this pull request Aug 4, 2024
…data compaction (#38742)

## problem

When enable_delete_when_cumu_compaction = true, multiple delete rowsets
may be merged into an empty data rowset through ordered data compaction,
causing the delete predicate to be lost.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged doing not-merge/2.1 reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants