[#2113] feat(pyClient): initial Gravitino Python client module #2676

zhaoyongjie · 2024-03-25T22:15:47Z

What changes were proposed in this pull request?

The more detailed introductions for the PR has been uploaded on the bilibili, please jump in there to get a quick view.

Follow the below command

Install the python-client in your local machine, notice that version of Python should be upper than 3.8.

$ cd clients/client-python
$ pip install -e .
$ pip install ipython (optional)
$ ipython (optional)

Run the Python terminal and happy coding.

In [1]: from gravitino import GravitinoClient
In [2]: client = GravitinoClient("http://localhost:8090")
In [3]: client.get_metalakes()
In [4]: metalake_demo = client.get_metalakes()[1]
In [5]: metalake_demo.catalog_hive.sales.customers.info()

I will follow next steps to refactor/implement the initial Python Client, also heads up @Lanznx was the initial contributor for the Python Client.

Remove requests since we are using really simple HTTP client to communicate between Python script and Gravitino, the http.client in Python3 is sufficient.
Add Unit test
Add codes linter
Provide an approach for uploading PyPI

Why are the changes needed?

The motivation of the PR intend to introduce an interactivity and exploratory user-experience between Python terminal and Gravitino Server.

Does this PR introduce any user-facing change?

N/A

How was this patch tested?

Unit tests will cover changes.

shaofengshi · 2024-03-27T09:21:15Z

Awesome @zhaoyongjie ! It is flexible, I can use the get_child method on each level, and also I can use the the quick way which is to use the child's name to directly accessing it, this is pretty user-friendly.

@zhaoyongjie one small thing, we expect to specify the "metalake_name" when initiating the gravitino client object, then after that, user doesn't need to aware the metalake much, directly list or operate on catalogs; We can refactor this later after this PR be merged. :)

Codes linter can be added later also; For the unit test, is a mock tool needed?

@coolderli @Lanznx Peidian and Brandon, please take a try as well. Thank you!

zhaoyongjie · 2024-03-27T10:09:09Z

one small thing, we expect to specify the "metalake_name" when initiating the gravitino client object, then after that, user doesn't need to aware the metalake much, directly list or operate on catalogs; We can refactor this later after this PR be merged. :)

It makes sense. I'll create another constructor in GravitinoClient might call initialize_matelake, which will have the same arguments as __init__ but also include a matelake_name parameter, after that will expose the new constructor method as a function in the top-level of client, the end-user should use GravitinoClient or GravitinoMatalake to initialize the instance of Gravitino.

Codes linter can be added later also; For the unit test, is a mock tool needed?

Given that the purpose of this utility is to serve as a client for Gravitino, using mock payloads might limit the scope for realistic testing. To be honest, I prefer integrating Python testing into Gravitino's integration tests and only maintaining a few basic unit tests within the utility itself.

coolderli · 2024-03-27T11:44:16Z

@zhaoyongjie Greate! Could you add some unit tests? The README.md may also need to be updated.

zhaoyongjie · 2024-03-27T12:10:40Z

Could you add some unit tests? The README.md may also need to be updated.

Sure, will add unit tests and updates the correlative document for the changes.

clients/client-python/requirements.txt

Add header for setup.py

This reverts commit ee4403d.

clients/client-python/gravitino/gravitino_client.py

xunliu · 2024-03-30T11:13:35Z

clients/client-python/gravitino/gravitino_client.py

+        protocol: str = "http",
+        port: int = 8090,


I think maybe we didn't need these parameter protocol and port.
Because we already have parameter host.

Depending on how the Client class is initialized, host, protocol, and port offer an alternative way to define the class

e.g.:

initialize_metalake("example-domain", "example-metalake", protocal="https", port=8080)

hi @zhaoyongjie thank you for your reply.
The host, protocol, and port parameters add to the complexity of the interface.
The host can be either http://localhost:8090 or localhost, which seems to be very flexible, but actually increases the cost of understanding. We just need to provide a robust host parameter.

clients/client-python/gravitino/typing.py

xunliu · 2024-04-01T01:45:10Z

clients/client-python/Makefile

Gravitino use Gradle to compile, so we didn't need this file.

It's for client test and build. Gradle might run it as well if you want.

I commit a PR #2753 to support Gradle to build and test Python client module, This way we can use the Gradle command in a unified way.

./gradlew build

./gradlew clean

./gradlew test

So, I sure we didn't need use this Makefile file.

clients/client-python/gravitino/gravitino_client.py

clients/client-python/gravitino/service.py

zhaoyongjie · 2024-04-01T09:44:22Z

so.....if I hadn't installed Gradle, there might be a block running such commands. And the other hand that Makefile wants to provide a way to create venv, ideally, Makefile might be an entry of the CI.

…

On Mon, Apr 1, 2024 at 11:35 AM Xun Liu ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ On clients/client-python/Makefile <#2676 (comment)> : I commit a PR #2753 <#2753> to support Gradle to build and test Python client module, This way we can use the Gradle command in a unified way. 1. ./gradlew build 2. ./gradlew clean 3. ./gradlew test — Reply to this email directly, view it on GitHub <#2676 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPMKUSNSP4TFG2SQZ67W43Y3ES75AVCNFSM6AAAAABFH2LGM2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSNZQHE4DINBWG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Best regards, Yongjie

xunliu · 2024-04-01T09:49:21Z

so.....if I hadn't installed Gradle, there might be a block running such commands. And the other hand that Makefile wants to provide a way to create venv, ideally, Makefile might be an entry of the CI.
…
On Mon, Apr 1, 2024 at 11:35 AM Xun Liu @.> wrote: @.* commented on this pull request. ------------------------------ On clients/client-python/Makefile <#2676 (comment)> : I commit a PR #2753 <#2753> to support Gradle to build and test Python client module, This way we can use the Gradle command in a unified way. 1. ./gradlew build 2. ./gradlew clean 3. ./gradlew test — Reply to this email directly, view it on GitHub <#2676 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPMKUSNSP4TFG2SQZ67W43Y3ES75AVCNFSM6AAAAABFH2LGM2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSNZQHE4DINBWG4 . You are receiving this because you were mentioned.Message ID: @.***>
-- Best regards, Yongjie

Developer didn't need install Gradle, Them only need have Java 8, 11, or 17.
When you execute ./gradlew command ant it will automatic download Gradle in template ./gradle directory.

clients/client-python/gravitino/service.py

clients/client-python/gravitino/__init__.py

coolderli · 2024-04-01T07:02:23Z

clients/client-python/tests/test_gravitino_client.py

+class TestGravitinoMetalake(unittest.TestCase):
+    def test_gravitino_metalake(self, *args):
+        metalake = gravitino_metalake("http://localhost:9000", "metalake_demo")
+        self.assertEqual(metalake.name, "metalake_demo")


We'd better assert the comment and audit

the class TestGravitinoMetalake(unittest.TestCase): inherit from unit test that is in Python standard library and removed pytest, aim to reduce the dependences for the beginning of a project.

BTW, pytest is also providing assertEqual statements.

xunliu

LGTM

… module (apache#2676)" This reverts commit 4411459.

zhaoyongjie changed the title ~~feat: initial Python Client~~ [WIP]feat: initial Python Client Mar 25, 2024

coolderli reviewed Mar 28, 2024

View reviewed changes

clients/client-python/requirements.txt Show resolved Hide resolved

Yongjie Zhao and others added 9 commits March 30, 2024 11:03

initial Python client

ae36106

wip

1c916da

wip

ba4abfb

wip

0be1a6e

wip

f31225e

Update setup.py

11665e8

Add header for setup.py

remove requests

18db11d

wip

33d0ae6

add unittests

e14faf1

zhaoyongjie changed the title ~~[WIP]feat: initial Python Client~~ feat: initial Python Client Mar 30, 2024

Yongjie Zhao added 5 commits March 30, 2024 11:21

Rm .gitignore

ee4403d

header of license

a9a327d

license

655bb61

Revert "Rm .gitignore"

c39e26a

This reverts commit ee4403d.

UT done

834c6f3

xunliu reviewed Apr 1, 2024

View reviewed changes

xunliu requested a review from shaofengshi April 1, 2024 09:59

xunliu changed the title ~~feat: initial Python Client~~ [#2113] feat(pyClient): initial Gravitino Python client module Apr 1, 2024

coolderli reviewed Apr 1, 2024

View reviewed changes

Yongjie Zhao added 3 commits April 1, 2024 13:33

rename methods

ff6850c

UTs

c19f12a

remove protocal and port

036655e

xunliu approved these changes Apr 2, 2024

View reviewed changes

xunliu merged commit 4411459 into apache:main Apr 2, 2024
19 checks passed

yuqi1129 added a commit to yuqi1129/gravitino that referenced this pull request Apr 3, 2024

Revert "[apache#2113] feat(pyClient): initial Gravitino Python client…

d3d4f3c

… module (apache#2676)" This reverts commit 4411459.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#2113] feat(pyClient): initial Gravitino Python client module #2676

[#2113] feat(pyClient): initial Gravitino Python client module #2676

zhaoyongjie commented Mar 25, 2024 •

edited by xunliu

Loading

shaofengshi commented Mar 27, 2024

zhaoyongjie commented Mar 27, 2024 •

edited

Loading

coolderli commented Mar 27, 2024

zhaoyongjie commented Mar 27, 2024

xunliu Mar 30, 2024

zhaoyongjie Apr 1, 2024

xunliu Apr 1, 2024 •

edited

Loading

xunliu Apr 1, 2024

zhaoyongjie Apr 1, 2024

xunliu Apr 1, 2024 •

edited

Loading

zhaoyongjie commented Apr 1, 2024 via email

xunliu commented Apr 1, 2024 •

edited

Loading

coolderli Apr 1, 2024

zhaoyongjie Apr 1, 2024

xunliu left a comment

[#2113] feat(pyClient): initial Gravitino Python client module #2676

[#2113] feat(pyClient): initial Gravitino Python client module #2676

Conversation

zhaoyongjie commented Mar 25, 2024 • edited by xunliu Loading

What changes were proposed in this pull request?

Follow the below command

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

shaofengshi commented Mar 27, 2024

zhaoyongjie commented Mar 27, 2024 • edited Loading

coolderli commented Mar 27, 2024

zhaoyongjie commented Mar 27, 2024

xunliu Mar 30, 2024

Choose a reason for hiding this comment

zhaoyongjie Apr 1, 2024

Choose a reason for hiding this comment

xunliu Apr 1, 2024 • edited Loading

Choose a reason for hiding this comment

xunliu Apr 1, 2024

Choose a reason for hiding this comment

zhaoyongjie Apr 1, 2024

Choose a reason for hiding this comment

xunliu Apr 1, 2024 • edited Loading

Choose a reason for hiding this comment

zhaoyongjie commented Apr 1, 2024 via email

xunliu commented Apr 1, 2024 • edited Loading

coolderli Apr 1, 2024

Choose a reason for hiding this comment

zhaoyongjie Apr 1, 2024

Choose a reason for hiding this comment

xunliu left a comment

Choose a reason for hiding this comment

zhaoyongjie commented Mar 25, 2024 •

edited by xunliu

Loading

zhaoyongjie commented Mar 27, 2024 •

edited

Loading

xunliu Apr 1, 2024 •

edited

Loading

xunliu Apr 1, 2024 •

edited

Loading

xunliu commented Apr 1, 2024 •

edited

Loading