HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS. #3322

jojochuang · 2021-08-23T09:33:08Z

Description of PR

This is a rebase of the patch file 11 attached to HDFS-6874.

The GETFILEBLOCKLOCATIONS is HCFS compatible. Add support of it to httpfs to makes it possible for more applications to run directly against HttpFS server.

Add GETFILEBLOCKLOCATIONS op support for httpfs server (HttpFSServer). Add the same for httpfs client (HttpFSFileSystem)
Let webhdfs client (WebHdfsFileSystem ) tries the new GETFILEBLOCKLOCATIONS op if the server supports it. Otherwise, fall back to the old GET_FILE_BLOCK_LOCATIONS op. The selection is cached so the second invocation doesn't need to trial and error again.

How was this patch tested?

Unit tests.

Change-Id: I9d0320cfc3286911784636254f2d99225e43ce8b

Change-Id: I520391820a98c87d3342983bdac144e5123a2e36

hadoop-yetus · 2021-08-23T15:24:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 44s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 10s		trunk passed
+1 💚	compile	4m 52s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	4m 38s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	3m 1s		trunk passed
+1 💚	javadoc	2m 10s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 39s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	6m 23s		trunk passed
+1 💚	shadedclient	14m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 20s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch failed.
-1 ❌	compile	4m 17s	/patch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt	hadoop-hdfs-project in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.
-1 ❌	javac	4m 17s	/patch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt	hadoop-hdfs-project in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.
-1 ❌	compile	4m 4s	/patch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-hdfs-project in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javac	4m 4s	/patch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-hdfs-project in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 7s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 4 new + 462 unchanged - 1 fixed = 466 total (was 463)
-1 ❌	mvnsite	0m 22s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch failed.
+1 💚	javadoc	1m 44s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 13s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	spotbugs	0m 21s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch failed.
+1 💚	shadedclient	16m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 21s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	231m 9s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
-1 ❌	unit	0m 36s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch failed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		349m 15s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/1/artifact/out/Dockerfile
GITHUB PR	#3322
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 267b991a4cbb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2e2192a`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/1/testReport/
Max. process+thread count	3413 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

amahussein

Thanks @jojochuang for rebasing the patch and addressing the most recent comments.

amahussein · 2021-08-23T16:28:06Z

...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java

+            offset, length);
+      }
+    } catch (RemoteException e) {
+      if (isGetFileBlockLocationsException(e)) {


Suggested change

if (isGetFileBlockLocationsException(e)) {

// parsing the exception is needed only if the client thinks the service is compatible

if (isServerHCFSCompatible && isGetFileBlockLocationsException(e)) {

...ject/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java

...ct/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java

...-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java

Change-Id: If6568f9d2227579f78f864794f3287ea0211c14a

hadoop-yetus · 2021-08-25T03:19:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 16s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 34s		trunk passed
+1 💚	compile	28m 33s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	23m 23s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	4m 36s		trunk passed
+1 💚	mvnsite	4m 37s		trunk passed
+1 💚	javadoc	3m 25s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	4m 12s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 34s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	18m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 11s		the patch passed
+1 💚	compile	27m 49s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	27m 49s		the patch passed
+1 💚	compile	24m 31s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	24m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 41s	/results-checkstyle-root.txt	root: The patch generated 8 new + 477 unchanged - 1 fixed = 485 total (was 478)
+1 💚	mvnsite	4m 19s		the patch passed
+1 💚	xml	0m 3s		The patch has no ill-formed XML file.
+1 💚	javadoc	3m 16s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 32s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 30s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	19m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 27s		hadoop-project in the patch passed.
+1 💚	unit	2m 50s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	321m 3s		hadoop-hdfs in the patch passed.
-1 ❌	unit	13m 36s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		579m 13s

Reason	Tests
Failed junit tests	hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
	hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/2/artifact/out/Dockerfile
GITHUB PR	#3322
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux c8350ad36873 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `af6b23d`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/2/testReport/
Max. process+thread count	2339 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java

...hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java

...ct/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java

...ject/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java

jojochuang · 2021-08-26T02:26:21Z

...ject/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java

+    Path path = new Path(pathStr);
+    DistributedFileSystem dfs = (DistributedFileSystem) FileSystem
+        .get(path.toUri(), TestHdfsHelper.getHdfsConf());
+    // Enable snapshot


no i don't understand this. it doesn't look like snapshot is used at all.

jojochuang · 2021-08-26T02:27:41Z

...ject/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java

+    Assert.assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+    BlockLocation[] locations1 =
+        dfs.getFileBlockLocations(new Path(file1), 0, 1);
+    Assert.assertNotNull(locations1);


it makes no sense to check nullity of locations1 using hdfs. The code doesn't change the file, so why check it?

jojochuang · 2021-08-26T02:27:49Z

...ject/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java

+    Assert.assertEquals(HttpURLConnection.HTTP_OK, conn1.getResponseCode());
+    BlockLocation[] locations2 =
+        dfs.getFileBlockLocations(new Path(file1), 0, 1);
+    Assert.assertNotNull(locations2);


Change-Id: I3f95a9839115f928de6b189783f283669b18e61a

hadoop-yetus · 2021-08-26T12:58:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 48s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 55s		trunk passed
+1 💚	compile	22m 49s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 31s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 57s		trunk passed
+1 💚	mvnsite	3m 50s		trunk passed
+1 💚	javadoc	2m 54s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 20s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 34s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	16m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 38s		the patch passed
+1 💚	compile	22m 12s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 12s		the patch passed
+1 💚	compile	19m 25s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	19m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 53s	/results-checkstyle-root.txt	root: The patch generated 9 new + 477 unchanged - 1 fixed = 486 total (was 478)
+1 💚	mvnsite	3m 45s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 54s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	3m 18s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	0m 30s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	17m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 27s		hadoop-project in the patch passed.
+1 💚	unit	2m 27s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	326m 15s		hadoop-hdfs in the patch passed.
-1 ❌	unit	14m 54s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt	hadoop-hdfs-httpfs in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		550m 53s

Reason	Tests
Failed junit tests	hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
	hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/3/artifact/out/Dockerfile
GITHUB PR	#3322
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux d33773cc2850 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `03f8bb7`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/3/testReport/
Max. process+thread count	2163 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

amahussein · 2021-09-01T15:01:39Z

Hey @jojochuang
Thanks for the recent fixes.
Are you still working on making changes to the PR or should I start doing a quick review?

Have you checked whether the failed unit tests are related to the changes?

[ERROR] Errors: 
[ERROR] org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem.testOperationDoAs[43](org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem)
[ERROR]   Run 1: TestHttpFSFWithSWebhdfsFileSystem>BaseTestHttpFSWith.testOperationDoAs:1391->BaseTestHttpFSWith.access$100:115->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2043 Â» SSL
[ERROR]   Run 2: TestHttpFSFWithSWebhdfsFileSystem>BaseTestHttpFSWith.testOperationDoAs:1391->BaseTestHttpFSWith.access$100:115->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2043 Â» SSL
[ERROR]   Run 3: TestHttpFSFWithSWebhdfsFileSystem>BaseTestHttpFSWith.testOperationDoAs:1391->BaseTestHttpFSWith.access$100:115->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2043 Â» SSL
[INFO] 
[ERROR] org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem.testOperation[43](org.apache.hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem)
[ERROR]   Run 1: TestHttpFSFWithSWebhdfsFileSystem>BaseTestHttpFSWith.testOperation:1380->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2043 Â» SSL
[ERROR]   Run 2: TestHttpFSFWithSWebhdfsFileSystem>BaseTestHttpFSWith.testOperation:1380->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2043 Â» SSL
[ERROR]   Run 3: TestHttpFSFWithSWebhdfsFileSystem>BaseTestHttpFSWith.testOperation:1380->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2043 Â» SSL
[INFO] 
[ERROR] org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem.testOperationDoAs[43](org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem)
[ERROR]   Run 1: TestHttpFSFileSystemLocalFileSystem>BaseTestHttpFSWith.testOperationDoAs:1391->BaseTestHttpFSWith.access$100:115->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2014 Â» ClassCast
[ERROR]   Run 2: TestHttpFSFileSystemLocalFileSystem>BaseTestHttpFSWith.testOperationDoAs:1391->BaseTestHttpFSWith.access$100:115->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2014 Â» ClassCast
[ERROR]   Run 3: TestHttpFSFileSystemLocalFileSystem>BaseTestHttpFSWith.testOperationDoAs:1391->BaseTestHttpFSWith.access$100:115->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2014 Â» ClassCast
[INFO] 
[ERROR] org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem.testOperation[43](org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem)
[ERROR]   Run 1: TestHttpFSFileSystemLocalFileSystem>BaseTestHttpFSWith.testOperation:1380->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2014 Â» ClassCast
[ERROR]   Run 2: TestHttpFSFileSystemLocalFileSystem>BaseTestHttpFSWith.testOperation:1380->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2014 Â» ClassCast
[ERROR]   Run 3: TestHttpFSFileSystemLocalFileSystem>BaseTestHttpFSWith.testOperation:1380->BaseTestHttpFSWith.operation:1351->BaseTestHttpFSWith.testGetFileBlockLocationsFallback:2014 Â» ClassCast

jojochuang · 2021-09-02T01:17:06Z

Hey Ahmed. Thanks a lot but I think this is going to require another few iterations.
Will ping you when I feel like it gets the quality I like.

ashutoshcipher · 2022-08-16T18:10:06Z

Hi @jojochuang. Thank you so much for working on this. Wanted to check if you are planning to take this forward or shall I plan to work on this.

amahussein · 2022-08-16T18:17:05Z

Hi @jojochuang. Thank you so much for working on this. Wanted to check if you are planning to take this forward or shall I plan to work on this.

Although the Jira pulls Github comments, this kind of question would better be posted on Jira.
A jira gets assigned to a developer while there might be multiple PRs to a single Jira ticket.

ashutoshcipher · 2022-08-16T18:21:40Z

Thanks @amahussein for pointing it out. Let me ask on JIRA.

aajisaka · 2022-11-21T07:41:17Z

Fixed by #4750

jojochuang added 2 commits August 20, 2021 16:45

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS

3c63213

Change-Id: I9d0320cfc3286911784636254f2d99225e43ce8b

Address review comments.

2e2192a

Change-Id: I520391820a98c87d3342983bdac144e5123a2e36

amahussein requested changes Aug 23, 2021

View reviewed changes

jojochuang commented Aug 24, 2021

View reviewed changes

...ct/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java Outdated Show resolved Hide resolved

jojochuang commented Aug 24, 2021

View reviewed changes

...-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java Outdated Show resolved Hide resolved

Address review comments, add more tests.

af6b23d

Change-Id: If6568f9d2227579f78f864794f3287ea0211c14a

jojochuang commented Aug 25, 2021

View reviewed changes

...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java Outdated Show resolved Hide resolved

jojochuang commented Aug 25, 2021

View reviewed changes

jojochuang commented Aug 26, 2021

View reviewed changes

Add tests, address review comments.

03f8bb7

Change-Id: I3f95a9839115f928de6b189783f283669b18e61a

aajisaka closed this Nov 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS. #3322

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS. #3322

jojochuang commented Aug 23, 2021

hadoop-yetus commented Aug 23, 2021

amahussein left a comment

amahussein Aug 23, 2021

hadoop-yetus commented Aug 25, 2021

jojochuang Aug 26, 2021

jojochuang Aug 26, 2021

jojochuang Aug 26, 2021

hadoop-yetus commented Aug 26, 2021

amahussein commented Sep 1, 2021

jojochuang commented Sep 2, 2021

ashutoshcipher commented Aug 16, 2022

amahussein commented Aug 16, 2022

ashutoshcipher commented Aug 16, 2022

aajisaka commented Nov 21, 2022

	if (isGetFileBlockLocationsException(e)) {
	// parsing the exception is needed only if the client thinks the service is compatible
	if (isServerHCFSCompatible && isGetFileBlockLocationsException(e)) {

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS. #3322

HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS. #3322

Conversation

jojochuang commented Aug 23, 2021

Description of PR

How was this patch tested?

hadoop-yetus commented Aug 23, 2021

amahussein left a comment

Choose a reason for hiding this comment

amahussein Aug 23, 2021

Choose a reason for hiding this comment

hadoop-yetus commented Aug 25, 2021

jojochuang Aug 26, 2021

Choose a reason for hiding this comment

jojochuang Aug 26, 2021

Choose a reason for hiding this comment

jojochuang Aug 26, 2021

Choose a reason for hiding this comment

hadoop-yetus commented Aug 26, 2021

amahussein commented Sep 1, 2021

jojochuang commented Sep 2, 2021

ashutoshcipher commented Aug 16, 2022

amahussein commented Aug 16, 2022

ashutoshcipher commented Aug 16, 2022

aajisaka commented Nov 21, 2022