feat: Implement services webhdfs (#1263)

* services: implement webhdfs's builder Signed-off-by: 蔡略 <[email protected]> * feat: implement webhdfs service Signed-off-by: 蔡略 <[email protected]> * refactor: fix unit tests Signed-off-by: 蔡略 <[email protected]> * refactor: add metadata for webhdfs 1. some documentation works 2. fix listing 3. try trouble shoot listing and writing, unsuccessful Signed-off-by: 蔡略 <[email protected]> * refactor: fix webhdfs read/write/delete Signed-off-by: 蔡略 <[email protected]> * refactor: implement blocking api to make build time checkings Signed-off-by: 蔡略 <[email protected]> * refactor: make clippy happy Signed-off-by: 蔡略 <[email protected]> * ci: complete webhdfs tests 1. add webhdfs test in ci, running simutaniously with hdfs test 2. add CSRF feature for webhdfs 3. finish documentation for webhdfs Signed-off-by: 蔡略 <[email protected]> * fix: correct default webhdfs port 1. default webhdfs port is 9870, controled by dfs.namenode.http-address Signed-off-by: 蔡略 <[email protected]> * ci: update webhdfs setup 1. set endpoint, hope this works Signed-off-by: 蔡略 <[email protected]> * refactor: webhdfs sync up with 0.27.0 Signed-off-by: 蔡略 <[email protected]> * refactor: check root in stat 1. don't run check root when building Backend, run it only when Backend::stat instead 2. move messages structs to `message.rs` 3. apply some advices in review Signed-off-by: 蔡略 <[email protected]> * refactor: make clippy happy Signed-off-by: 蔡略 <[email protected]> * fix: fix dead lock in checking root Signed-off-by: 蔡略 <[email protected]> * fix: doc test in webhdfs Signed-off-by: 蔡略 <[email protected]> * refactor: make cargo msrv clippy happy Signed-off-by: 蔡略 <[email protected]> * ci: fix webhdfs integated test Signed-off-by: 蔡略 <[email protected]> * ci: recorrect test command line Signed-off-by: 蔡略 <[email protected]> * Fix CI Signed-off-by: Xuanwo <[email protected]> * User runner as webhdfs user Signed-off-by: Xuanwo <[email protected]> * ci: change webhdfs test root Signed-off-by: 蔡略 <[email protected]> * typo: fix incorrect documentation 1. fix incorrect documentation in webhdfs 2. remove unnecessary language environments in webhdfs CI Signed-off-by: 蔡略 <[email protected]> * refactor: remove username and doas 1. it does not work at all, implement kerberos later Signed-off-by: 蔡略 <[email protected]> * fix: typo fix Signed-off-by: 蔡略 <[email protected]> --------- Signed-off-by: 蔡略 <[email protected]> Signed-off-by: Xuanwo <[email protected]> Co-authored-by: Xuanwo <[email protected]>
apache · Feb 7, 2023 · fb1189e · fb1189e · github-actions · Feb 7, 2023
1 parent 360b150
commit fb1189e
Show file tree

Hide file tree

Showing 12 changed files with 1,145 additions and 2 deletions.
diff --git a/.env.example b/.env.example
@@ -73,3 +73,8 @@ OPENDAL_MEMCACHED_ROOT=/
 OPENDAL_WEBDAV_TEST=false
 OPENDAL_WEBDAV_ROOT=/tmp/opendal/
 OPENDAL_WEBDAV_ENDPOINT=http://127.0.0.1:8080
+# webhfds
+OPENDAL_WEBHDFS_TEST=false
+OPENDAL_WEBHDFS_ROOT=/tmp/opendal/
+OPENDAL_WEBHDFS_ENDPOINT=http://127.0.0.1:9870
+OPENDAL_WEBHDFS_DELEGATION=<delegation>
diff --git a/.github/workflows/service_test_webhdfs.yml b/.github/workflows/service_test_webhdfs.yml
@@ -0,0 +1,42 @@
+name: Service Test WebHDFS
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+    paths-ignore:
+      - "docs/**"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}
+  cancel-in-progress: true
+
+jobs:
+  hdfs:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        hdfs-version: ["2.10.1", "3.2.3", "3.3.2"]
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Setup-hdfs env
+        uses: beyondstorage/setup-hdfs@master
+        with:
+          hdfs-version: ${{ matrix.hdfs-version }}
+
+      - uses: Swatinem/rust-cache@v2
+      - name: Test
+        shell: bash
+        continue-on-error: true
+        run: cargo test webhdfs -- --show-output
+        env:
+          RUST_BACKTRACE: full
+          RUST_LOG: debug
+          OPENDAL_WEBHDFS_TEST: on
+          OPENDAL_WEBHDFS_ROOT: /tmp/opendal/
+          OPENDAL_WEBHDFS_USER: runner
+          OPENDAL_WEBHDFS_ENDPOINT: http://${{ env.HDFS_NAMENODE_HTTP_ADDR }}
diff --git a/README.md b/README.md
@@ -32,6 +32,7 @@
 - [rocksdb](https://opendal.databend.rs/opendal/services/struct.Rocksdb.html): [RocksDB](http://rocksdb.org/) services support.
 - [s3](https://opendal.databend.rs/opendal/services/struct.S3.html): [AWS S3](https://aws.amazon.com/s3/) alike services.
 - [webdav](https://opendal.databend.rs/opendal/services/struct.Webdav.html): [WebDAV](https://datatracker.ietf.org/doc/html/rfc4918) Service Support.
+- [webhdfs](https://opendal.databend.rs/opendal/services/struct.Webhdfs.html): [WebHDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html) Service Support.
 
 ## Features
 

diff --git a/src/raw/rps.rs b/src/raw/rps.rs
@@ -16,11 +16,11 @@ use http::Request;
 
 use crate::*;
 
-/// Reply fro `create` operation
+/// Reply for `create` operation
 #[derive(Debug, Clone, Default)]
 pub struct RpCreate {}
 
-/// Reply fro `delete` operation
+/// Reply for `delete` operation
 #[derive(Debug, Clone, Default)]
 pub struct RpDelete {}
 

diff --git a/src/scheme.rs b/src/scheme.rs
@@ -73,6 +73,8 @@ pub enum Scheme {
     S3,
     /// [webdav][crate::services::Webdav]: WebDAV support.
     Webdav,
+    /// [webhdfs][crate::services::WebHdfs]: WebHDFS RESTful API Services
+    Webhdfs,
     /// Custom that allow users to implement services outside of OpenDAL.
     ///
     /// # NOTE
@@ -133,6 +135,7 @@ impl FromStr for Scheme {
             "s3" => Ok(Scheme::S3),
             "oss" => Ok(Scheme::Oss),
             "webdav" => Ok(Scheme::Webdav),
+            "webhdfs" => Ok(Scheme::Webhdfs),
             _ => Ok(Scheme::Custom(Box::leak(s.into_boxed_str()))),
         }
     }
@@ -167,6 +170,7 @@ impl From<Scheme> for &'static str {
             Scheme::S3 => "s3",
             Scheme::Oss => "oss",
             Scheme::Webdav => "webdav",
+            Scheme::Webhdfs => "webhdfs",
             Scheme::Custom(v) => v,
         }
     }

diff --git a/src/services/mod.rs b/src/services/mod.rs
@@ -86,3 +86,6 @@ pub use s3::S3;
 
 mod webdav;
 pub use webdav::Webdav;
+
+mod webhdfs;
+pub use webhdfs::Webhdfs;