Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](agg function) incorrect result of map agg #39743

Merged
merged 1 commit into from
Aug 23, 2024
Merged

[fix](agg function) incorrect result of map agg #39743

merged 1 commit into from
Aug 23, 2024

Conversation

mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Aug 21, 2024

Proposed changes

In the function void add(const Field& key_, const Field& value), it should not return while the key exists in the set.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mrhhsg
Copy link
Member Author

mrhhsg commented Aug 21, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Aug 21, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/aggregate_functions/aggregate_function_map.h
@@ -18,7 +18,6 @@
#pragma once

#include <parallel_hashmap/phmap.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'parallel_hashmap/phmap.h' file not found [clang-diagnostic-error]

#include <parallel_hashmap/phmap.h>
         ^

@doris-robot
Copy link

TPC-H: Total hot run time: 38681 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1da90daf0624bf1ef4f107fc5531c5aac5e901ae, data reload: false

------ Round 1 ----------------------------------
q1	17848	4509	4366	4366
q2	2069	216	211	211
q3	11512	937	1308	937
q4	10535	773	791	773
q5	7801	2894	2832	2832
q6	284	161	164	161
q7	1046	679	658	658
q8	9400	2134	2150	2134
q9	7336	6559	6590	6559
q10	7103	2317	2312	2312
q11	501	287	279	279
q12	438	263	272	263
q13	17981	3077	3037	3037
q14	322	271	266	266
q15	591	519	521	519
q16	528	418	417	417
q17	997	716	695	695
q18	7436	6734	6936	6734
q19	1514	1027	1004	1004
q20	698	351	359	351
q21	3910	3130	3201	3130
q22	1158	1043	1053	1043
Total cold run time: 111008 ms
Total hot run time: 38681 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4412	4335	4356	4335
q2	416	307	299	299
q3	2909	2693	2717	2693
q4	1930	1656	1780	1656
q5	5682	5733	5710	5710
q6	238	149	159	149
q7	2256	1840	1891	1840
q8	3317	3454	3449	3449
q9	8937	8909	8906	8906
q10	3597	3356	3403	3356
q11	658	531	563	531
q12	879	690	695	690
q13	15874	3231	3182	3182
q14	340	314	307	307
q15	589	531	523	523
q16	513	473	464	464
q17	1843	1525	1556	1525
q18	8220	7860	7806	7806
q19	2394	1515	1637	1515
q20	2143	1948	1939	1939
q21	5742	5573	5494	5494
q22	1204	1079	1107	1079
Total cold run time: 74093 ms
Total hot run time: 57448 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199063 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1da90daf0624bf1ef4f107fc5531c5aac5e901ae, data reload: false

query1	1319	925	874	874
query2	6427	1988	1986	1986
query3	10685	4048	4054	4048
query4	59720	27510	23361	23361
query5	5587	739	721	721
query6	453	219	210	210
query7	5815	340	340	340
query8	515	437	435	435
query9	9142	2564	2516	2516
query10	580	356	340	340
query11	18076	15100	15560	15100
query12	196	146	142	142
query13	1650	448	451	448
query14	11468	8011	8084	8011
query15	252	215	197	197
query16	7582	521	498	498
query17	1158	627	656	627
query18	2083	357	358	357
query19	310	174	173	173
query20	152	139	140	139
query21	248	161	141	141
query22	4577	4402	4467	4402
query23	35036	34111	34156	34111
query24	6108	3000	2984	2984
query25	576	444	432	432
query26	709	183	192	183
query27	1718	349	320	320
query28	3739	2167	2132	2132
query29	766	458	467	458
query30	235	201	194	194
query31	1042	894	887	887
query32	110	79	82	79
query33	519	360	367	360
query34	941	537	519	519
query35	897	775	783	775
query36	1083	964	992	964
query37	168	117	111	111
query38	4004	3941	3866	3866
query39	1553	1492	1454	1454
query40	236	159	157	157
query41	141	136	137	136
query42	138	118	118	118
query43	538	490	489	489
query44	1148	805	806	805
query45	227	199	196	196
query46	1130	802	786	786
query47	1938	1838	1874	1838
query48	439	340	347	340
query49	923	598	608	598
query50	865	478	489	478
query51	7223	7238	7098	7098
query52	116	109	118	109
query53	295	228	229	228
query54	624	521	517	517
query55	96	91	87	87
query56	342	324	323	323
query57	1263	1144	1130	1130
query58	302	304	309	304
query59	3092	2848	2699	2699
query60	365	339	333	333
query61	152	152	151	151
query62	803	701	691	691
query63	266	235	265	235
query64	3391	1918	1868	1868
query65	3294	3217	3203	3203
query66	951	693	698	693
query67	15395	15319	15054	15054
query68	4485	616	610	610
query69	477	343	327	327
query70	1266	1183	1175	1175
query71	393	326	328	326
query72	6416	2433	2171	2171
query73	785	366	367	366
query74	9228	8831	8861	8831
query75	3476	2746	2750	2746
query76	1769	1030	1022	1022
query77	655	456	454	454
query78	9755	9103	9141	9103
query79	1094	583	568	568
query80	943	648	634	634
query81	566	268	264	264
query82	260	168	159	159
query83	238	217	214	214
query84	280	107	102	102
query85	901	370	363	363
query86	340	336	330	330
query87	4399	4346	4277	4277
query88	3196	2556	2552	2552
query89	428	329	325	325
query90	1892	246	243	243
query91	161	131	128	128
query92	90	78	77	77
query93	1101	548	556	548
query94	794	324	322	322
query95	402	308	311	308
query96	610	297	293	293
query97	3234	3118	3126	3118
query98	240	228	232	228
query99	1596	1322	1320	1320
Total cold run time: 314362 ms
Total hot run time: 199063 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1da90daf0624bf1ef4f107fc5531c5aac5e901ae, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.05	0.04
query3	0.23	0.05	0.05
query4	1.65	0.08	0.09
query5	0.53	0.49	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.55	0.49	0.50
query10	0.54	0.55	0.54
query11	0.17	0.13	0.13
query12	0.16	0.13	0.13
query13	0.63	0.59	0.59
query14	0.76	0.79	0.79
query15	0.86	0.83	0.85
query16	0.38	0.38	0.38
query17	1.05	1.06	1.00
query18	0.22	0.21	0.21
query19	1.91	1.74	1.84
query20	0.02	0.01	0.01
query21	15.40	0.67	0.67
query22	3.68	7.02	2.47
query23	18.21	1.38	1.32
query24	2.10	0.22	0.23
query25	0.17	0.09	0.08
query26	0.28	0.18	0.18
query27	0.09	0.08	0.08
query28	13.31	1.03	1.00
query29	12.61	3.40	3.38
query30	0.44	0.25	0.24
query31	2.80	0.40	0.41
query32	3.24	0.49	0.50
query33	2.98	2.96	3.02
query34	16.83	4.38	4.37
query35	4.44	4.45	4.49
query36	0.68	0.51	0.52
query37	0.22	0.19	0.18
query38	0.17	0.18	0.17
query39	0.06	0.05	0.06
query40	0.19	0.17	0.16
query41	0.12	0.08	0.07
query42	0.08	0.08	0.07
query43	0.07	0.06	0.06
Total cold run time: 109.17 s
Total hot run time: 31.92 s

@mrhhsg
Copy link
Member Author

mrhhsg commented Aug 22, 2024

run buildall

@yiguolei yiguolei added dev/2.1.x usercase Important user case type label labels Aug 22, 2024
@mrhhsg
Copy link
Member Author

mrhhsg commented Aug 22, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38690 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5ca934aec0d0b009d6a4ea2db5afa89d4960a0f6, data reload: false

------ Round 1 ----------------------------------
q1	18283	4712	4356	4356
q2	2065	215	206	206
q3	11738	998	1177	998
q4	10416	712	847	712
q5	7787	2927	2866	2866
q6	268	159	157	157
q7	1041	663	640	640
q8	9378	2155	2155	2155
q9	7213	6627	6669	6627
q10	7085	2282	2195	2195
q11	505	265	279	265
q12	447	261	261	261
q13	18709	3050	3078	3050
q14	316	285	250	250
q15	547	532	520	520
q16	534	408	415	408
q17	1012	710	704	704
q18	7436	6827	6963	6827
q19	1480	1101	1019	1019
q20	691	346	353	346
q21	4044	3061	3216	3061
q22	1163	1075	1067	1067
Total cold run time: 112158 ms
Total hot run time: 38690 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4435	4329	4502	4329
q2	404	299	306	299
q3	2923	2726	2599	2599
q4	1965	1771	1763	1763
q5	5684	5665	5700	5665
q6	255	158	153	153
q7	2264	1789	1795	1789
q8	3308	3439	3442	3439
q9	8916	8823	8819	8819
q10	3602	3320	3448	3320
q11	625	539	535	535
q12	873	682	692	682
q13	16443	3225	3250	3225
q14	348	330	328	328
q15	597	517	518	517
q16	507	457	459	457
q17	1817	1553	1542	1542
q18	8200	7994	7844	7844
q19	2157	1631	1692	1631
q20	2171	1936	1948	1936
q21	5600	5448	5506	5448
q22	1220	1138	1082	1082
Total cold run time: 74314 ms
Total hot run time: 57402 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198501 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5ca934aec0d0b009d6a4ea2db5afa89d4960a0f6, data reload: false

query1	1296	897	904	897
query2	6470	1904	1988	1904
query3	10788	4325	4090	4090
query4	60168	25237	23254	23254
query5	5554	747	747	747
query6	463	214	209	209
query7	5802	333	340	333
query8	526	441	437	437
query9	9027	2564	2519	2519
query10	523	336	315	315
query11	18013	15039	15670	15039
query12	180	131	128	128
query13	1620	455	457	455
query14	10530	7514	8038	7514
query15	264	206	218	206
query16	7462	488	490	488
query17	1165	624	630	624
query18	2069	355	346	346
query19	323	199	180	180
query20	153	142	140	140
query21	249	149	148	148
query22	4674	4381	4419	4381
query23	35367	34345	34529	34345
query24	6094	3095	2955	2955
query25	575	427	423	423
query26	681	193	185	185
query27	1679	320	315	315
query28	4003	2178	2171	2171
query29	704	462	463	462
query30	241	211	194	194
query31	1036	886	875	875
query32	105	80	84	80
query33	545	370	366	366
query34	942	534	519	519
query35	897	793	783	783
query36	1109	958	971	958
query37	166	107	109	107
query38	4088	3954	3911	3911
query39	1510	1488	1465	1465
query40	252	162	159	159
query41	139	138	139	138
query42	134	118	116	116
query43	546	511	504	504
query44	1187	798	786	786
query45	222	197	192	192
query46	1123	804	768	768
query47	1931	1855	1831	1831
query48	439	342	341	341
query49	925	597	599	597
query50	874	476	480	476
query51	7247	6967	7159	6967
query52	119	107	113	107
query53	312	232	227	227
query54	646	536	510	510
query55	93	91	90	90
query56	346	326	321	321
query57	1232	1145	1122	1122
query58	313	324	312	312
query59	2957	2882	3039	2882
query60	363	353	341	341
query61	152	154	153	153
query62	800	692	717	692
query63	270	237	233	233
query64	2931	1897	1855	1855
query65	3279	3209	3215	3209
query66	1021	689	675	675
query67	15379	15120	15450	15120
query68	4513	606	615	606
query69	475	334	327	327
query70	1174	1106	1127	1106
query71	453	327	318	318
query72	2619	2167	2103	2103
query73	751	361	367	361
query74	9190	9044	8870	8870
query75	3447	2777	2750	2750
query76	1805	1075	1009	1009
query77	656	445	447	445
query78	10307	9060	9132	9060
query79	1093	570	553	553
query80	912	645	688	645
query81	493	265	263	263
query82	316	162	165	162
query83	242	216	221	216
query84	273	99	99	99
query85	843	365	363	363
query86	353	333	318	318
query87	4560	4326	4327	4326
query88	3300	2549	2523	2523
query89	430	327	327	327
query90	1928	234	237	234
query91	157	128	126	126
query92	90	79	75	75
query93	1118	550	549	549
query94	797	333	328	328
query95	400	302	315	302
query96	628	297	299	297
query97	3259	3148	3103	3103
query98	237	233	222	222
query99	1596	1360	1323	1323
Total cold run time: 310708 ms
Total hot run time: 198501 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.07 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5ca934aec0d0b009d6a4ea2db5afa89d4960a0f6, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.08	0.09
query5	0.54	0.50	0.49
query6	1.14	0.73	0.74
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.55	0.49	0.49
query10	0.55	0.54	0.54
query11	0.18	0.12	0.12
query12	0.16	0.12	0.12
query13	0.63	0.60	0.59
query14	0.77	0.79	0.79
query15	0.85	0.82	0.85
query16	0.38	0.37	0.37
query17	0.99	0.99	1.03
query18	0.22	0.22	0.21
query19	1.90	1.79	1.82
query20	0.01	0.02	0.01
query21	15.40	0.66	0.65
query22	4.50	7.77	1.53
query23	18.30	1.49	1.33
query24	2.04	0.24	0.24
query25	0.16	0.08	0.08
query26	0.28	0.18	0.18
query27	0.09	0.09	0.09
query28	13.32	1.02	1.02
query29	12.66	3.38	3.38
query30	0.44	0.25	0.24
query31	2.81	0.41	0.41
query32	3.24	0.49	0.49
query33	2.97	2.98	3.06
query34	17.03	4.38	4.47
query35	4.50	4.48	4.48
query36	0.66	0.51	0.50
query37	0.21	0.18	0.18
query38	0.18	0.18	0.17
query39	0.06	0.06	0.06
query40	0.18	0.15	0.15
query41	0.12	0.06	0.07
query42	0.08	0.07	0.07
query43	0.07	0.07	0.06
Total cold run time: 110.27 s
Total hot run time: 31.07 s

@yiguolei yiguolei merged commit 2048fbc into apache:master Aug 23, 2024
27 of 30 checks passed
@mrhhsg mrhhsg mentioned this pull request Aug 23, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 24, 2024
@mrhhsg
[fix](agg function) incorrect result of map agg(#39743) (#39854)
8cf6c6a 
## Proposed changes

pick #39743
@mrhhsg mrhhsg deleted the fix_map_agg branch August 25, 2024 02:17
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
@mrhhsg @dataroaring
[fix](agg function) incorrect result of map agg (#39743)
b998365 
## Proposed changes

In the function `void add(const Field& key_, const Field& value)`, it
should not return while the key exists in the set.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@yiguolei yiguolei yiguolei approved these changes

@jacktengg jacktengg jacktengg approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mrhhsg @doris-robot @yiguolei @jacktengg @dataroaring