Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](cloud) Fix missing privilege to storage vault after restarting FE #40260

Merged
merged 1 commit into from
Sep 4, 2024
Merged

[fix](cloud) Fix missing privilege to storage vault after restarting FE #40260

merged 1 commit into from
Sep 4, 2024

Conversation

gavinchou
Copy link
Contributor

The previous implement forgets to build storageVaultPrivTable (in-memory) after loading auth information from image, which means the privileges are persisted but unable to use after restarting FE.

Note: a new image will be generated after FE restarts or a checkpoint is triggered,
the newly generated will be used when FE starts.

@gavinchou
[fix](cloud) Fix missing privilege to storage vault after restarting FE
8945743 
The previous implement forgets to build `storageVaultPrivTable` (in-memory)
after loading auth information from image, which means the privileges
are persisted but unable to use after restarting FE.

Note: a new image will be generated after FE restarts.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38391 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 89457434d9ed59a5ca5a61dca07e1d59364b53d0, data reload: false

------ Round 1 ----------------------------------
q1	17838	4516	4452	4452
q2	2677	178	177	177
q3	10998	1124	1126	1124
q4	10277	726	758	726
q5	8215	2867	2890	2867
q6	238	141	145	141
q7	991	627	612	612
q8	9565	2084	2035	2035
q9	7289	6509	6508	6508
q10	7026	2252	2172	2172
q11	440	235	240	235
q12	402	226	224	224
q13	17792	3100	3073	3073
q14	286	243	230	230
q15	531	499	498	498
q16	584	532	525	525
q17	967	675	699	675
q18	7435	6788	6839	6788
q19	1405	1014	1103	1014
q20	673	341	321	321
q21	4325	2959	3088	2959
q22	1133	1035	1040	1035
Total cold run time: 111087 ms
Total hot run time: 38391 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4365	4405	4336	4336
q2	376	282	268	268
q3	2900	2683	2650	2650
q4	1926	1661	1629	1629
q5	5389	5387	5416	5387
q6	219	133	135	133
q7	2084	1723	1748	1723
q8	3248	3373	3345	3345
q9	8472	8370	8444	8370
q10	3445	3183	3210	3183
q11	591	508	482	482
q12	789	625	621	621
q13	12008	3097	3134	3097
q14	302	273	274	273
q15	511	486	475	475
q16	591	570	556	556
q17	1761	1502	1494	1494
q18	7761	7296	7744	7296
q19	1850	1542	1562	1542
q20	2029	1801	1817	1801
q21	5407	5292	5221	5221
q22	1117	1060	1062	1060
Total cold run time: 67141 ms
Total hot run time: 54942 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188421 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 89457434d9ed59a5ca5a61dca07e1d59364b53d0, data reload: false

query1	919	375	370	370
query2	6453	2086	1992	1992
query3	6656	206	216	206
query4	29935	23354	23239	23239
query5	4218	530	487	487
query6	257	162	163	162
query7	4588	304	290	290
query8	277	205	213	205
query9	8492	2481	2484	2481
query10	459	289	285	285
query11	17864	15013	15048	15013
query12	153	99	96	96
query13	1635	380	383	380
query14	9182	6946	7035	6946
query15	223	175	169	169
query16	7446	467	462	462
query17	1576	560	544	544
query18	1766	284	277	277
query19	191	142	145	142
query20	115	107	105	105
query21	210	124	101	101
query22	4565	4125	4113	4113
query23	34306	33575	33248	33248
query24	11122	2905	2793	2793
query25	643	384	372	372
query26	1182	151	167	151
query27	2682	277	276	276
query28	7403	2061	2047	2047
query29	780	418	403	403
query30	305	165	151	151
query31	978	756	782	756
query32	100	55	55	55
query33	764	284	284	284
query34	992	476	477	476
query35	842	741	693	693
query36	1071	910	935	910
query37	160	94	94	94
query38	3995	3802	3856	3802
query39	1458	1405	1404	1404
query40	198	117	113	113
query41	49	47	49	47
query42	119	98	95	95
query43	520	472	491	472
query44	1230	751	738	738
query45	201	179	168	168
query46	1113	759	756	756
query47	1893	1793	1797	1793
query48	373	296	300	296
query49	1086	464	427	427
query50	802	415	407	407
query51	7297	7173	7029	7029
query52	98	87	87	87
query53	249	186	178	178
query54	848	446	443	443
query55	79	74	75	74
query56	264	256	249	249
query57	1164	1102	1076	1076
query58	248	220	233	220
query59	3024	3084	2758	2758
query60	320	264	271	264
query61	99	99	104	99
query62	856	659	643	643
query63	216	180	177	177
query64	4450	674	656	656
query65	3196	3136	3142	3136
query66	1385	343	342	342
query67	15625	15519	15227	15227
query68	3560	575	571	571
query69	410	278	274	274
query70	1117	1133	1095	1095
query71	334	276	275	275
query72	6400	4110	4031	4031
query73	742	327	329	327
query74	9151	9010	8960	8960
query75	3374	2650	2682	2650
query76	1820	994	1047	994
query77	489	346	318	318
query78	9547	9759	10423	9759
query79	1008	526	533	526
query80	690	511	507	507
query81	497	244	240	240
query82	235	140	147	140
query83	169	147	149	147
query84	217	76	82	76
query85	893	292	288	288
query86	307	299	297	297
query87	4385	4259	4220	4220
query88	3084	2317	2340	2317
query89	437	285	286	285
query90	1904	186	192	186
query91	119	100	102	100
query92	64	48	50	48
query93	1033	543	529	529
query94	854	298	293	293
query95	349	252	252	252
query96	580	268	271	268
query97	3193	3097	3030	3030
query98	223	205	197	197
query99	1496	1271	1275	1271
Total cold run time: 281646 ms
Total hot run time: 188421 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 89457434d9ed59a5ca5a61dca07e1d59364b53d0, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.70	0.08	0.08
query5	0.50	0.50	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.54	0.49	0.49
query10	0.55	0.54	0.53
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.60	0.58
query14	2.10	2.11	2.10
query15	0.90	0.82	0.82
query16	0.38	0.37	0.37
query17	1.00	1.00	0.99
query18	0.21	0.19	0.20
query19	1.77	1.77	1.85
query20	0.01	0.01	0.00
query21	15.42	0.66	0.67
query22	4.10	6.67	2.11
query23	18.27	1.42	1.27
query24	2.15	0.23	0.23
query25	0.15	0.08	0.08
query26	0.26	0.17	0.18
query27	0.07	0.07	0.08
query28	13.22	1.00	1.00
query29	12.62	3.39	3.42
query30	0.25	0.05	0.06
query31	2.91	0.39	0.38
query32	3.23	0.48	0.47
query33	2.95	3.04	3.01
query34	17.07	4.32	4.37
query35	4.42	4.41	4.43
query36	0.67	0.50	0.49
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.13
query41	0.09	0.05	0.05
query42	0.07	0.05	0.06
query43	0.04	0.04	0.04
Total cold run time: 110.63 s
Total hot run time: 32.26 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 approved these changes Sep 4, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gavinchou gavinchou merged commit 553de2d into apache:master Sep 4, 2024
29 of 33 checks passed
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

PR approved by at least one committer and no changes requested.

gavinchou added a commit that referenced this pull request Sep 5, 2024
@gavinchou
[fix](cloud) Fix missing privilege to storage vault after restarting …
261dc20 
…FE (#40260)

The previous implement forgets to build `storageVaultPrivTable`
(in-memory) after loading auth information from image, which means the
privileges are persisted but unable to use after restarting FE.

Note: a new image will be generated after FE restarts or a checkpoint is
triggered,
the newly generated will be used when FE starts.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deardeng deardeng deardeng left review comments

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@gavinchou @doris-robot @deardeng @liaoxin01 @Yukang-Lian