HADOOP-19226: [ABFS][FNSOverBlob] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #6944

anujmodi2021 · 2024-07-16T09:26:13Z

Description of PR

This is the second PR in the series of work done under Parent Jira: HADOOP-19179
Jira for this Patch: https://issues.apache.org/jira/browse/HADOOP-19226

Scope of this task is to implement Blob Endpoint Rest APIs offered by Azure Storage into AbfsBlobClient which will be another child of AbfsClient just like AbfsDfsClient.

The blob endpoint support will remain "Unsupported" until the whole code is checked-in and well tested.

Production Code Changes

AbfsBlobClient will be another child of AbfsClient similar to AbfsDfsClient but will have Blob API implementations.
AzureBlobFileSystemStore will now have a AbfsClientHandler which will hold both the clients so that store can call onto the one it needs to talk. Some HDFS API's store function definition is changed to make it common for both DFS and Blob client so that proper abstraction can be established.

Test Code Changes

Existing tests are modified to work with new abstracted design.
Fns-Blob Support is too immature to add tests around Blob Endpoint usage or AbfsBlobClient itself at this point. Follow up PRs on FnsOverBlob Support will add new tests once the whole implementation is complete. Till then users won't be allowed to configure blob endpoint.
Test to verify that FS init will fail with Blob Endpoint added.
Test Suite was run on DFS Endpoint to make sure the original driver works seamlessly and undisturbed.

How was this patch tested?

Existing test suite was ran on DFS Endpoint only. All the failures reported are known.
Metric related tests are fixed in the the #6847

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

hadoop-yetus · 2024-08-29T07:15:35Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	48m 26s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 10s		trunk passed
+1 💚	shadedclient	37m 49s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	38m 9s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 26s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 25s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 7s		the patch passed
+1 💚	shadedclient	37m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 24s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		137m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/7/artifact/out/Dockerfile
GITHUB PR	#6944
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname	Linux 09a1d87f9190 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2702954`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/7/testReport/
Max. process+thread count	551 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-08-29T07:26:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 56s		trunk passed
+1 💚	compile	0m 27s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 20s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 26s		trunk passed
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 45s		trunk passed
+1 💚	shadedclient	20m 19s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	20m 32s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 18s		the patch passed
+1 💚	compile	0m 16s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 41s		the patch passed
+1 💚	shadedclient	20m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 57s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		89m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/8/artifact/out/Dockerfile
GITHUB PR	#6944
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname	Linux 53ed2ea69360 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f167931`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/8/testReport/
Max. process+thread count	552 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021 · 2024-09-18T06:11:29Z

Hi @saikatroy038
Kindly inviting you to review this PR.

For some background work.
This is the second PR in the series of work we are doing to deprecate WASB Driver.
This PR has changes on top of: #6879

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

rakeshadr

Thanks @anujmodi2021 for taking care this feature. Good work! Added a few comments, pls address it.

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientHandler.java

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FSOperationType.java

.../src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AppendRequestParameters.java

...hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperationType.java

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

...adoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClientHandler.java

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java

anujmodi2021 · 2024-10-01T05:58:51Z

HI @saikatroy038 @rakeshadr
I have addressed your comments on the PR...
Kindly review the latest iteration.
Thanks for helping

hadoop-yetus · 2024-10-01T08:16:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	45m 19s		trunk passed
+1 💚	compile	0m 40s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 6s		trunk passed
+1 💚	shadedclient	39m 33s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	39m 56s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 33s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 33s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 12s		the patch passed
+1 💚	shadedclient	39m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 33s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		139m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/11/artifact/out/Dockerfile
GITHUB PR	#6944
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname	Linux 0a93de588139 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `097162e`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/11/testReport/
Max. process+thread count	725 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/11/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

partial review; its late and I need to eat...

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java

...op-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HttpHeaderConfigurations.java

.../src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AppendRequestParameters.java

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

steveloughran · 2024-10-04T19:09:10Z

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

+  @Override
+  public AbfsRestOperation breakLease(final String path,
+      TracingContext tracingContext) throws AzureBlobFileSystemException {
+    final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();


This operation is called a lot and so is duplicated code. Is there a way to make it a bit leaner?
e.g some method

add(List<AbfsHttpHeader> l, string key, string val) { l.add(new AbfsHttpHeader(key, val);

then you could do a terser sequence

I think I missed this comment. This seems like a good suggestion...
I don't have much context on terser sequence but I will surely learn more on how it works and how this will be helpful in making code cleaner.

I will take this up in future JIRAs related to FNS-Blob support.
Tracking Jira: https://issues.apache.org/jira/browse/HADOOP-19333

hadoop-yetus · 2024-10-08T11:37:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 16s		trunk passed
+1 💚	compile	0m 25s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 20s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 22s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 47s		trunk passed
+1 💚	shadedclient	20m 16s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	20m 29s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 20s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 20s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 14s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 5 new + 3 unchanged - 0 fixed = 8 total (was 3)
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 20s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 47s		the patch passed
+1 💚	shadedclient	20m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 0s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 26s		The patch does not generate ASF License warnings.
		88m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/13/artifact/out/Dockerfile
GITHUB PR	#6944
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname	Linux 7dfd48f048e5 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `109de38`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/13/testReport/
Max. process+thread count	551 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/13/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-10-08T11:40:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 34s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	0m 42s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 12s		trunk passed
+1 💚	shadedclient	44m 27s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	44m 49s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 10s		the patch passed
+1 💚	shadedclient	40m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 26s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		146m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/12/artifact/out/Dockerfile
GITHUB PR	#6944
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname	Linux ea02fb72439f 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4dff5fd`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/12/testReport/
Max. process+thread count	728 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/12/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-10-20T12:01:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 28s		trunk passed
-1 ❌	compile	0m 24s	/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-azure in trunk failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
+1 💚	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 43s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 11s		trunk passed
+1 💚	shadedclient	41m 13s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	41m 35s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
-1 ❌	compile	0m 19s	/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-azure in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌	javac	0m 19s	/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-azure in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 21s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 5 new + 3 unchanged - 0 fixed = 8 total (was 3)
+1 💚	mvnsite	0m 33s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 21s		the patch passed
+1 💚	shadedclient	39m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 25s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		141m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/14/artifact/out/Dockerfile
GITHUB PR	#6944
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle markdownlint
uname	Linux b8cda5bc9dd4 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cd0690b`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/14/testReport/
Max. process+thread count	686 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/14/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-11-14T08:31:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 20s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 32s		trunk passed
+1 💚	mvnsite	0m 43s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 13s		trunk passed
+1 💚	shadedclient	41m 55s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	42m 17s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 34s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 9 new + 7 unchanged - 0 fixed = 16 total (was 7)
+1 💚	mvnsite	0m 33s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 27s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 11s		the patch passed
+1 💚	shadedclient	40m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 27s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		141m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/15/artifact/out/Dockerfile
GITHUB PR	#6944
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle markdownlint
uname	Linux 8a1fbe1fceda 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `35fac84`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/15/testReport/
Max. process+thread count	706 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6944/15/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021 · 2024-11-14T09:24:24Z

Hi @steveloughran @rakeshadr
Thanks for reviewing the PR.
I have addressed your comments

Please let me know if this looks good now.

rakeshadr · 2024-11-24T16:44:25Z

@anujmodi2021 There are a few unresolved comments and I could see a "Changes requested", please check those. Thanks!

rakeshadr

+1 LGTM, please resolve pending comments.

steveloughran

+1 from me too

…b Endpoint for AbfsBlobClient (apache#6944) Contributed by Anuj Modi

Blob Endpoint Support

a16f7da

github-actions bot added trunk TOOLS ABFS labels Jul 16, 2024

Anuj Modi added 2 commits August 20, 2024 22:10

Merge branch 'trunk' into HADOOP-19226_blobEndpointImp

67f0946

Javadocs

eb0cfa8

anujmodi2021 marked this pull request as ready for review August 21, 2024 06:07

anmolanmol1234 reviewed Aug 21, 2024

View reviewed changes

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java Outdated Show resolved Hide resolved

Anuj Modi added 5 commits August 21, 2024 23:11

Yetus Checks

2203ab8

Resolved Comments

c32419d

Javadoc Errors

9d4f5dc

ClientHandler Changed

2702954

Added Tests

f167931

saikatroy038 reviewed Sep 19, 2024

View reviewed changes

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java Outdated Show resolved Hide resolved

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java Outdated Show resolved Hide resolved

rakeshadr reviewed Sep 19, 2024

View reviewed changes

Anuj Modi added 4 commits September 29, 2024 20:52

Addressing Comments

9b95fef

Merge branch 'trunk' into HADOOP-19226_blobEndpointImp

c6b1c2b

Yetus Checks

ff51790

Fixing Failing Tests

097162e

anujmodi2021 mentioned this pull request Oct 4, 2024

[ABFS][FNSOverBlob] Making AbfsClient Abstract for supporting both DFS and Blob Endpoint #6846

Closed

4 tasks

apache deleted a comment from hadoop-yetus Oct 4, 2024

steveloughran requested changes Oct 4, 2024

View reviewed changes

Anuj Modi added 2 commits October 8, 2024 02:12

Added More Javadocs

4dff5fd

Added More Loggings'

109de38

anujmodi2021 self-assigned this Oct 20, 2024

Blob Endpoint Markdown File

cd0690b

anujmodi2021 added 3 commits November 12, 2024 03:12

Merge branch 'trunk' into HADOOP-19226_blobEndpointImp

f346247

Addressing Comments and CheckStyle Fixes'

8b92bf0

markdown file changes

35fac84

rakeshadr requested review from steveloughran and rakeshadr November 24, 2024 16:42

rakeshadr approved these changes Nov 25, 2024

View reviewed changes

steveloughran approved these changes Nov 25, 2024

View reviewed changes

rakeshadr merged commit 65a5bf3 into apache:trunk Nov 26, 2024
4 checks passed

anujmodi2021 added a commit to ABFSDriver/AbfsHadoop that referenced this pull request Dec 27, 2024

HADOOP-19226: [ABFS][FNSOverBlob] Implementing Azure Rest APIs on Blo…

1c3c937

…b Endpoint for AbfsBlobClient (apache#6944) Contributed by Anuj Modi

anujmodi2021 mentioned this pull request Dec 27, 2024

HADOOP-19226: [ABFS][FNSOverBlob][Backport 3.4] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #7253

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19226: [ABFS][FNSOverBlob] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #6944

HADOOP-19226: [ABFS][FNSOverBlob] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #6944

anujmodi2021 commented Jul 16, 2024 •

edited

Loading

hadoop-yetus commented Aug 29, 2024

hadoop-yetus commented Aug 29, 2024

anujmodi2021 commented Sep 18, 2024

rakeshadr left a comment

anujmodi2021 commented Oct 1, 2024

hadoop-yetus commented Oct 1, 2024

steveloughran left a comment

steveloughran Oct 4, 2024

anujmodi2021 Nov 26, 2024 •

edited

Loading

hadoop-yetus commented Oct 8, 2024

hadoop-yetus commented Oct 8, 2024

hadoop-yetus commented Oct 20, 2024

hadoop-yetus commented Nov 14, 2024

anujmodi2021 commented Nov 14, 2024

rakeshadr commented Nov 24, 2024

rakeshadr left a comment

steveloughran left a comment

HADOOP-19226: [ABFS][FNSOverBlob] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #6944

HADOOP-19226: [ABFS][FNSOverBlob] Implementing Azure Rest APIs on Blob Endpoint for AbfsBlobClient #6944

Conversation

anujmodi2021 commented Jul 16, 2024 • edited Loading

Description of PR

The blob endpoint support will remain "Unsupported" until the whole code is checked-in and well tested.

Production Code Changes

Test Code Changes

How was this patch tested?

hadoop-yetus commented Aug 29, 2024

hadoop-yetus commented Aug 29, 2024

anujmodi2021 commented Sep 18, 2024

rakeshadr left a comment

Choose a reason for hiding this comment

anujmodi2021 commented Oct 1, 2024

hadoop-yetus commented Oct 1, 2024

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran Oct 4, 2024

Choose a reason for hiding this comment

anujmodi2021 Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

hadoop-yetus commented Oct 8, 2024

hadoop-yetus commented Oct 8, 2024

hadoop-yetus commented Oct 20, 2024

hadoop-yetus commented Nov 14, 2024

anujmodi2021 commented Nov 14, 2024

rakeshadr commented Nov 24, 2024

rakeshadr left a comment

Choose a reason for hiding this comment

steveloughran left a comment

Choose a reason for hiding this comment

anujmodi2021 commented Jul 16, 2024 •

edited

Loading

anujmodi2021 Nov 26, 2024 •

edited

Loading