feat: Enable custom keyword arguments in Atlas Proxy Client #248
Conversation
mgorsk1
requested review from
allisonsuarez,
bolkedebruin,
dikshathakur3119 and
jinhyukchang
as code owners
|
cc @dechoma |
Signed-off-by: mgorsk1 <[email protected]>
mgorsk1
force-pushed
the
feature/kwargs_in_proxy_client
branch
from
886bb1e to
5ed686e
Compare
| @@ -84,6 +85,10 @@ class Config: | |||
| # format to be used in tables | |||
| WATERMARK_DATE_FORMATS = ['%Y%m%d'] | |||
|
|
|||
| # Custom kwargs that will be passed to proxy client. Can be used to fine-tune parameters like timeout | |||
| # or num of retries | |||
| PROXY_CLIENT_KWARGS: Dict = dict() | |||
There was a problem hiding this comment.
do you have example of this config? and is it possible to be leverage by other proxies?
There was a problem hiding this comment.
It definitely can be leveraged by different proxies. Two examples:
- In neo4j proxy there are settings (https://github.com/amundsen-io/amundsenmetadatalibrary/blob/master/metadata_service/proxy/neo4j_proxy.py#L54) that clearly are not set by this method https://github.com/amundsen-io/amundsenmetadatalibrary/blob/master/metadata_service/proxy/__init__.py#L39. I think that currently users don't have any way of fine tuning them, but they could be easily passed under this new configuration option.
- Some time ago we made a POC with GCP Data Catalog as Amundsen proxy and GCP client has completely different set of args in comparison to what Neo4j, Atlas or Neptune expect (like path to GCP service account key file). We used similar solution back then and with some new proxies it might come in handy.
In our case we started small with PROXY_CLIENT_KWARGS = dict(timeout=int(os.getenv('ATLAS_REQUEST_TIMEOUT_SEC', 30))) but there are couple of more params we could add (num_of_retries being one of them)
|
@feng-tao ok to accept & merge? |
|
Thanks ! @feng-tao |
| @@ -70,7 +70,8 @@ def __init__(self, *, | |||
| user: str = 'admin', | |||
| password: str = '', | |||
| encrypted: bool = False, | |||
| validate_ssl: bool = False) -> None: | |||
| validate_ssl: bool = False, | |||
| client_kwargs: dict = dict()) -> None: | |||
There was a problem hiding this comment.
should use None as default argument.
|
@mgorsk1 could you take a look at this issue (amundsen-io/amundsen#881) ? Looking at the code, if people doesn't define the config, I think it will break other proxy (neo4j, neptune etc)? |
Summary of Changes
This PR adds a possibility to pass custom kwargs to Atlas proxy client. It be used to fine-tune parameters like timeout or num of retries. Since every client might accept different kind of extra parameters it might be easier to just have custom dictionary complementing
Tests
Documentation
CheckList
Make sure you have checked all steps below to ensure a timely review.
make test