HADOOP-19226: [ABFS][FNSOverBlob][Backport 3.4] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #7253

anujmodi2021 · 2024-12-27T12:01:43Z

This is the second PR in the series of work done under Parent Jira: HADOOP-19179
Jira for this Patch: https://issues.apache.org/jira/browse/HADOOP-19226

Scope of this task is to implement Blob Endpoint Rest APIs offered by Azure Storage into AbfsBlobClient which will be another child of AbfsClient just like AbfsDfsClient.

Original PR was merged to trunk and had enough bake time: #6944

Cherry-picked commit: 65a5bf3

How was this patch tested?

Test suite was ran.
Results added below

…b Endpoint for AbfsBlobClient (apache#6944) Contributed by Anuj Modi

anujmodi2021 · 2024-12-27T12:02:00Z

:::: AGGREGATED TEST RESULT ::::

============================================================
HNS-OAuth

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 651, Failures: 0, Errors: 0, Skipped: 82
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 651, Failures: 0, Errors: 0, Skipped: 34
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 635, Failures: 0, Errors: 0, Skipped: 274
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 651, Failures: 0, Errors: 0, Skipped: 84
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

hadoop-yetus · 2024-12-27T13:18:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ branch-3.4 Compile Tests _
+1 💚	mvninstall	21m 28s		branch-3.4 passed
+1 💚	compile	0m 23s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 20s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 21s		branch-3.4 passed
+1 💚	mvnsite	0m 26s		branch-3.4 passed
+1 💚	javadoc	0m 25s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 22s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 42s		branch-3.4 passed
+1 💚	shadedclient	18m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 18s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 11s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 8 new + 5 unchanged - 0 fixed = 13 total (was 5)
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 16s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 41s		the patch passed
+1 💚	shadedclient	18m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 8s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		75m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7253/1/artifact/out/Dockerfile
GITHUB PR	#7253
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle markdownlint
uname	Linux 91352226ef09 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / `1c3c937`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7253/1/testReport/
Max. process+thread count	553 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7253/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021 · 2025-01-09T03:35:09Z

@mukund-thakur @saikatroy038
Request to review this backport PR.

Thanks

steveloughran · 2025-01-09T11:20:25Z

does this need HADOOP-19382. [ABFS][FnsOverBlob] Test Fix for ITestAzureBlobFileSystemInitAndCreate failure #7269? if so, I've just upvoted that, so can be combined with this
backports do not need review, if you are happy with it -do the merge

anujmodi2021 · 2025-01-10T05:50:30Z

does this need HADOOP-19382. [ABFS][FnsOverBlob] Test Fix for ITestAzureBlobFileSystemInitAndCreate failure #7269? if so, I've just upvoted that, so can be combined with this

backports do not need review, if you are happy with it -do the merge

Makes sense steve, thanks for heads up, will combine both and merge post validations.

…emInitAndCreate failure (apache#7269) contributed by Anuj Modi (anujmodi)

hadoop-yetus · 2025-01-10T08:22:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	19m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ branch-3.4 Compile Tests _
+1 💚	mvninstall	39m 1s		branch-3.4 passed
+1 💚	compile	0m 39s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 35s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 31s		branch-3.4 passed
+1 💚	mvnsite	0m 39s		branch-3.4 passed
+1 💚	javadoc	0m 38s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 33s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 4s		branch-3.4 passed
+1 💚	shadedclient	37m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 8 new + 5 unchanged - 0 fixed = 13 total (was 5)
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 3s		the patch passed
+1 💚	shadedclient	37m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 16s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		146m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7253/2/artifact/out/Dockerfile
GITHUB PR	#7253
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle markdownlint
uname	Linux 716b1287b263 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / `6ec46f1`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7253/2/testReport/
Max. process+thread count	587 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7253/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

HADOOP-19226: [ABFS][FNSOverBlob] Implementing Azure Rest APIs on Blo…

1c3c937

…b Endpoint for AbfsBlobClient (apache#6944) Contributed by Anuj Modi

anujmodi2021 requested a review from mukund-thakur January 9, 2025 03:34

HADOOP-19382. [ABFS][FnsOverBlob] Test Fix for ITestAzureBlobFileSyst…

6ec46f1

…emInitAndCreate failure (apache#7269) contributed by Anuj Modi (anujmodi)

anujmodi2021 merged commit 0006912 into apache:branch-3.4 Jan 15, 2025
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19226: [ABFS][FNSOverBlob][Backport 3.4] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #7253

HADOOP-19226: [ABFS][FNSOverBlob][Backport 3.4] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #7253

anujmodi2021 commented Dec 27, 2024

anujmodi2021 commented Dec 27, 2024

hadoop-yetus commented Dec 27, 2024

anujmodi2021 commented Jan 9, 2025

steveloughran commented Jan 9, 2025

anujmodi2021 commented Jan 10, 2025

hadoop-yetus commented Jan 10, 2025

HADOOP-19226: [ABFS][FNSOverBlob][Backport 3.4] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #7253

HADOOP-19226: [ABFS][FNSOverBlob][Backport 3.4] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #7253

Conversation

anujmodi2021 commented Dec 27, 2024

Original PR was merged to trunk and had enough bake time: #6944

How was this patch tested?

anujmodi2021 commented Dec 27, 2024

============================================================ HNS-OAuth

============================================================ HNS-SharedKey

============================================================ NonHNS-SharedKey

============================================================ AppendBlob-HNS-OAuth

hadoop-yetus commented Dec 27, 2024

anujmodi2021 commented Jan 9, 2025

steveloughran commented Jan 9, 2025

anujmodi2021 commented Jan 10, 2025

hadoop-yetus commented Jan 10, 2025

============================================================
HNS-OAuth

============================================================
HNS-SharedKey

============================================================
NonHNS-SharedKey

============================================================
AppendBlob-HNS-OAuth