feat(ingest/tableau): support ingestion of access roles #11157

haeniya · 2024-08-13T06:02:04Z

Description

This PR enables the optional ingestion of access roles for the Tableau ingestion. To do this, we fetch the group permissions from the Tableau API, optionally transform it to the users needs and add them as roles to the Tableau asset. Currently, it is only implemented for Workbooks but could be extended to other assets as well.

The configuration in the recipe could look something like this:

source:
  type: tableau
  config:
    # Coordinates
    connect_uri: https://tableau.example.com

    access_role_ingestion:
      enable_workbooks: True
      role_prefix: "R-Tableau-"
      group_substring_start: 29
      role_description: "IAM Role required to access this Tableau asset."
      displayed_capabilities: ["Read", "Write", "Delete"]
      request_url: "https://iam.example.com/accessRequest?role=$ROLE_NAME"
      group_name_pattern:
        allow: [ "^.*_Consumer$" ]

    # Credentials
    username: "${TABLEAU_USER}"
    password: "${TABLEAU_PASSWORD}"

sink:
  type: "datahub-rest"
  config:
    server: "http://localhost:8080"

The access aspect was only available for datasets before. That's why I added it to containers now. The "Access Management" tab should only be visible in the UI if there's an access aspect associated with the container.

Let me know what you think and I will add some tests for this.

Thanks.

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

…ure/tableau-ingestion-access-permissions

coderabbitai · 2024-08-13T06:02:10Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

metadata-ingestion/src/datahub/ingestion/source/tableau.py

haeniya · 2024-08-20T12:03:46Z

metadata-ingestion/src/datahub/ingestion/source/tableau.py

@@ -264,8 +273,9 @@ def make_tableau_client(self, site: str) -> Server:
                },
            )

-            # From https://stackoverflow.com/a/50159273/5004662.
-            server._session.trust_env = False
+            if not self.session_trust_env:


This was something we discovered along the way. It seems to disable / bypass proxy settings which is not what we want in our case. That's why we made it configurable. Let me know what you think.

haeniya · 2024-09-10T07:14:59Z

If anyone has an idea regarding the failing tests, please let me know. They mostly don't seem related to my changes. I see linting issues in unity source and failing tests for other sources. Am I missing something?

…ure/tableau-ingestion-access-permissions

sid-acryl

Please update tableau connector documentation also for groups are getting ingested as role in DataHub with example of group name in Tableau and corresponding role in DataHub.

metadata-ingestion/src/datahub/ingestion/source/tableau/tableau.py

sid-acryl · 2024-09-11T10:44:35Z

metadata-ingestion/src/datahub/ingestion/source/tableau/tableau.py

+        description="Use this property if you only need a substring of the group in Tableau to create the role used in the request_url.",
+    )
+
+    role_description: str = Field(


Do we need a map here as there might be multiple roles ?

For us it's currently enough to just have one generic description for all the roles. This could be extended to enable mapping between maybe a regex pattern for a role name and a role description in the future. But maybe we wait with this until we see the need for it, WDYT?

haeniya · 2024-09-11T13:27:39Z

Please update tableau connector documentation also for groups are getting ingested as role in DataHub with example of group name in Tableau and corresponding role in DataHub.

@sid-acryl do you mean adding a new section in https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/tableau/tableau_pre.md that explains the new feature of ingesting access roles with some examples?

sid-acryl · 2024-09-11T16:17:15Z

Please update tableau connector documentation also for groups are getting ingested as role in DataHub with example of group name in Tableau and corresponding role in DataHub.

@sid-acryl do you mean adding a new section in https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/tableau/tableau_pre.md that explains the new feature of ingesting access roles with some examples?

Yup and please also update the Concept Mapping

haeniya · 2024-09-12T10:16:36Z

I'm currently fixing the tests and adding a new one - just fyi.

sid-acryl

Lets verify if we need any update in Quickstart guide of tableau (https://github.com/datahub-project/datahub/tree/master/docs/quick-ingestion-guides/tableau)

sid-acryl · 2024-09-13T05:26:50Z

metadata-ingestion/docs/sources/tableau/tableau_pre.md

+  config:
+    server: "http://localhost:8080"
+```
+Assuming you have groups in Tableau with names like `AB_XY00-Tableau-Access_A_123_PROJECT_XY_Consumer` or `AB_XY00-Tableau-Access_A_123_PROJECT_XY_Analyst` and corresponding IAM roles like `AR-Tableau-PROJECT_XY_Consumer` or `AR-Tableau-PROJECT_XY_Analyst`.


Please move this assumption to above the recipe and mention how to map this information in recipe to achieve correct transformation.

@sid-acryl I moved it above the recipe and improved the wording. Let me know what you think.

haeniya · 2024-09-16T14:01:41Z

Lets verify if we need any update in Quickstart guide of tableau (https://github.com/datahub-project/datahub/tree/master/docs/quick-ingestion-guides/tableau)

I'd say we don't need to update this part.

sid-acryl · 2024-10-07T10:17:43Z

What is the motivation behind below config properties

role_prefix

group_substring_start

group_substring_end

As I tried to describe it in the docs (https://github.com/datahub-project/datahub/pull/11157/files#diff-9a69800355baee2a7495dab637dd6bb46ba1f3be3a6b73bbc7bc70c22b9aa07fR78-R81) those are for transforming the group names in Tableau to what is used as role name in the request link. In our case we have IAM roles that are related to each Tableau group but the names are not exactly the same. These properties are currently enough for us to transform the group names coming from Tableau to their corresponding IAM roles that we want to use in the request link.

Are you referring to AWS IAM roles when you mention IAM?

haeniya · 2024-10-07T10:24:55Z

What is the motivation behind below config properties

role_prefix

group_substring_start

group_substring_end

As I tried to describe it in the docs (https://github.com/datahub-project/datahub/pull/11157/files#diff-9a69800355baee2a7495dab637dd6bb46ba1f3be3a6b73bbc7bc70c22b9aa07fR78-R81) those are for transforming the group names in Tableau to what is used as role name in the request link. In our case we have IAM roles that are related to each Tableau group but the names are not exactly the same. These properties are currently enough for us to transform the group names coming from Tableau to their corresponding IAM roles that we want to use in the request link.

Are you referring to AWS IAM roles when you mention IAM?

Could be any Identity & Access Management tool. We're using Sailpoint but it doesn't really matter. It doesn't even need to be an IAM tool. Those fields are just there to transform the Tableau groups to what is used in the request link as $ROLE_NAME. Could be any tool. Maybe we remove IAM in the docs to avoid confusion.

sid-acryl

Let's remove the term IAM.
Is it possible for you to test it with other Identity or Cloud Provider (google, azure) ?

metadata-ingestion/src/datahub/ingestion/source/tableau/tableau.py

sid-acryl · 2024-10-07T10:32:54Z

metadata-ingestion/src/datahub/ingestion/source/tableau/tableau.py

@@ -298,6 +308,56 @@ def make_tableau_client(self, site: str) -> Server:
            ) from e


+class AccessRolesIngestionConfig(ConfigModel):
+    enable_workbooks: bool = Field(


enable would be sufficient here.

The idea behind enable_workbooks is that we can add more entities in the future like views or dashboards. We could then just add a new config property enable_dashboards or enable_views. WDYT?

metadata-ingestion/src/datahub/ingestion/source/tableau/tableau.py

sid-acryl · 2024-10-07T11:23:02Z

metadata-ingestion/src/datahub/ingestion/source/tableau/tableau.py

+
+    request_url: str = Field(
+        default="",
+        description="Request URL that is added to the role for requesting access. "


As per code it looks like $ROLE_NAME is required to substitute the role name in request_url.

Request URL that is added to access role metadata. The request URL must include $ROLE_NAME, For example 'https://example-host.com/order/$ROLE_NAME'

No, $ROLE_NAME shouldn't be required. If it's not present it just won't be replaced. Or where do you see an issue?

haeniya · 2024-10-07T12:56:00Z

Let's remove the term IAM. Is it possible for you to test it with other Identity or Cloud Provider (google, azure) ?

Changed various parts according to your feedback. Please have a look at the remaining comments. To your question - no, unfortunately, I don't have the option to test with other IAM providers. But as already said, it doesn't matter what IAM tool is used. The way access requests work in Datahub currently is that there is a link in the Access Management tab of an asset that redirects the user some external page. My changes are only related to this link that is created for each Tableau group. In theory, any link can be created if the role name can be derived from the Tableau group.

hsheth2 · 2024-10-08T18:20:35Z

Hey folks, I should've jumped in a bit earlier on this PR.

I'm good with

adding access aspect to containers
all the relevant UI changes
fetching user/group info for workbooks from the tableau API

The piece I am worried about is this - this convention feels a bit too specific to your setup.

Assuming you have groups in Tableau with names such as `AB_XY00-Tableau-Access_A_123_PROJECT_XY_Consumer` or `AB_XY00-Tableau-Access_A_123_PROJECT_XY_Analyst` and corresponding roles (e.g. in an IAM tool) like `AR-Tableau-PROJECT_XY_Consumer` or `AR-Tableau-PROJECT_XY_Analyst`.
Using the recipe below, would filter the groups that end with "_Consumer" and transform the group names into the corresponding roles.
With `group_start_index` and `group_end_index` you can define a substring of the group name to be used as role name and `role_prefix` can be used to add a prefix to the generated role name.
The role names then act as a substitute of `$ROLE_NAME` in the `request_url` and result in access request URLs like `https://iam.example.com/accessRequest?role=AR-Tableau-PROJECT_XY_Consumer`, for example.

If we can conclusively say that this is a standard convention for linking AWS / other IAM setups with Tableau, and that other DataHub users will also benefit from this specific setup, then I'm good with it. But I haven't really heard of this setup from other folks thus far.

At the same time, I do want to figure out how we can make this work for you. I'd personally be in favor of putting the user/group info in the container's custom properties. That way, you could still build a transformer that can convert custom properties into access aspects.

…p as custom property

haeniya · 2024-10-14T11:12:57Z

Hey folks, I should've jumped in a bit earlier on this PR.

I'm good with

adding access aspect to containers

all the relevant UI changes

fetching user/group info for workbooks from the tableau API

The piece I am worried about is this - this convention feels a bit too specific to your setup.
Assuming you have groups in Tableau with names such as `AB_XY00-Tableau-Access_A_123_PROJECT_XY_Consumer` or `AB_XY00-Tableau-Access_A_123_PROJECT_XY_Analyst` and corresponding roles (e.g. in an IAM tool) like `AR-Tableau-PROJECT_XY_Consumer` or `AR-Tableau-PROJECT_XY_Analyst`.
Using the recipe below, would filter the groups that end with "_Consumer" and transform the group names into the corresponding roles.
With `group_start_index` and `group_end_index` you can define a substring of the group name to be used as role name and `role_prefix` can be used to add a prefix to the generated role name.
The role names then act as a substitute of `$ROLE_NAME` in the `request_url` and result in access request URLs like `https://iam.example.com/accessRequest?role=AR-Tableau-PROJECT_XY_Consumer`, for example.
If we can conclusively say that this is a standard convention for linking AWS / other IAM setups with Tableau, and that other DataHub users will also benefit from this specific setup, then I'm good with it. But I haven't really heard of this setup from other folks thus far.

At the same time, I do want to figure out how we can make this work for you. I'd personally be in favor of putting the user/group info in the container's custom properties. That way, you could still build a transformer that can convert custom properties into access aspects.

Hi @hsheth2, as discussed offline, I refactored the code and removed the access aspect and role creation again. The ingestion now only sets the Tableau groups as custom properties. Access and role creation will be done in a transformer. Please review the changes again and let me know what you think. Thanks.

metadata-ingestion/docs/sources/tableau/tableau_pre.md

metadata-ingestion/src/datahub/ingestion/source/tableau/tableau.py

hsheth2 · 2024-10-18T15:43:59Z

@haeniya feels like we're pretty close here :)

…as json

hsheth2 · 2024-10-23T22:45:42Z

@haeniya looks like the tests are failing

FAILED tests/integration/tableau/test_tableau_ingest.py::test_group_ingestion - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (tableau): 1 validation error for TableauConfig
group_ingestion
  extra fields not permitted (type=value_error.extra)

…//github.com/haeniya/datahub into feature/tableau-ingestion-access-permissions

…ect#11157) Co-authored-by: Yanik Häni <[email protected]> Co-authored-by: Harshal Sheth <[email protected]>

Yanik Häni added 4 commits August 12, 2024 07:48

add access aspect to container, permission ingestion wip

6903459

fix linting issues

f9e1d7e

Merge branch 'master' of https://github.com/haeniya/datahub into feat…

7edb40d

…ure/tableau-ingestion-access-permissions

cleanup

ee8eb3d

github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX community-contribution PR or Issue raised by member(s) of DataHub Community labels Aug 13, 2024

vercel bot had a problem deploying to Preview August 13, 2024 06:10 Failure

githendrik reviewed Aug 20, 2024

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/tableau.py Outdated Show resolved Hide resolved

make trust_env property configurable, not hard-coded, minor text change

33c9a26

haeniya commented Aug 20, 2024

View reviewed changes

vercel bot had a problem deploying to Preview August 20, 2024 12:09 Failure

hsheth2 requested a review from sid-acryl September 10, 2024 03:45

Merge branch 'master' of https://github.com/haeniya/datahub into feat…

a237aa6

…ure/tableau-ingestion-access-permissions

vercel bot deployed to Preview September 10, 2024 07:53 View deployment

sid-acryl suggested changes Sep 11, 2024

View reviewed changes

pr feedback implemented: field descriptions and minor refactorings

a866287

add some documentation with examples to the tableau source docs

9d9c96a

vercel bot deployed to Preview September 12, 2024 07:39 View deployment

sid-acryl suggested changes Sep 13, 2024

View reviewed changes

Yanik Häni added 2 commits September 16, 2024 14:34

add unit test for new functionality, improve docs

9414cfb

fix linting issues

556a060

vercel bot deployed to Preview September 16, 2024 13:37 View deployment

fix frontend linting issues

1219699

vercel bot deployed to Preview October 1, 2024 05:59 View deployment

haeniya requested a review from sid-acryl October 2, 2024 05:46

sid-acryl suggested changes Oct 7, 2024

View reviewed changes

field description changes, minor refactorings according to pr feedback

d6fca5e

vercel bot deployed to Preview October 7, 2024 13:03 View deployment

pr feedback: remove access aspect and role creation and only add grou…

4dca4d5

…p as custom property

vercel bot deployed to Preview October 14, 2024 11:24 View deployment

hsheth2 reviewed Oct 15, 2024

View reviewed changes

Yanik Häni added 2 commits October 21, 2024 13:07

pr review: remove displayed capabilities prop and output permissions …

1ba89e6

…as json

fix linting issues

76b8651

vercel bot deployed to Preview October 21, 2024 11:36 View deployment

hsheth2 approved these changes Oct 21, 2024

View reviewed changes

hsheth2 added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Oct 21, 2024

hsheth2 changed the title ~~feat(tableau-ingestion): Enable optional ingestion of access roles~~ feat(tableau): support ingestion of access roles Oct 21, 2024

Merge branch 'master' into feature/tableau-ingestion-access-permissions

3975605

vercel bot deployed to Preview October 21, 2024 22:25 View deployment

Yanik Häni and others added 3 commits October 24, 2024 07:54

fix integration tests

af988bd

Merge branch 'feature/tableau-ingestion-access-permissions' of https:…

7dcadad

…//github.com/haeniya/datahub into feature/tableau-ingestion-access-permissions

Merge branch 'master' into feature/tableau-ingestion-access-permissions

45949bd

vercel bot deployed to Preview October 24, 2024 07:14 View deployment

hsheth2 changed the title ~~feat(tableau): support ingestion of access roles~~ feat(ingest/tableau): support ingestion of access roles Oct 24, 2024

hsheth2 merged commit 7c8dba4 into datahub-project:master Oct 24, 2024
78 of 80 checks passed

sleeperdeep pushed a commit to sleeperdeep/datahub that referenced this pull request Dec 17, 2024

feat(ingest/tableau): support ingestion of access roles (datahub-proj…

7c5f062

…ect#11157) Co-authored-by: Yanik Häni <[email protected]> Co-authored-by: Harshal Sheth <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingest/tableau): support ingestion of access roles #11157

feat(ingest/tableau): support ingestion of access roles #11157

haeniya commented Aug 13, 2024

coderabbitai bot commented Aug 13, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

haeniya Aug 20, 2024

haeniya commented Sep 10, 2024

sid-acryl left a comment

sid-acryl Sep 11, 2024

haeniya Sep 11, 2024

haeniya commented Sep 11, 2024

sid-acryl commented Sep 11, 2024

haeniya commented Sep 12, 2024

sid-acryl left a comment

sid-acryl Sep 13, 2024

haeniya Sep 16, 2024

haeniya commented Sep 16, 2024 •

edited

Loading

sid-acryl commented Oct 7, 2024

haeniya commented Oct 7, 2024 •

edited

Loading

sid-acryl left a comment

sid-acryl Oct 7, 2024

haeniya Oct 7, 2024 •

edited

Loading

sid-acryl Oct 7, 2024

haeniya Oct 7, 2024

haeniya commented Oct 7, 2024 •

edited

Loading

hsheth2 commented Oct 8, 2024

haeniya commented Oct 14, 2024

hsheth2 commented Oct 18, 2024

hsheth2 commented Oct 23, 2024

feat(ingest/tableau): support ingestion of access roles #11157

feat(ingest/tableau): support ingestion of access roles #11157

Conversation

haeniya commented Aug 13, 2024

Description

Checklist

coderabbitai bot commented Aug 13, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Choose a reason for hiding this comment

haeniya commented Sep 10, 2024

sid-acryl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haeniya commented Sep 11, 2024

sid-acryl commented Sep 11, 2024

haeniya commented Sep 12, 2024

sid-acryl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haeniya commented Sep 16, 2024 • edited Loading

sid-acryl commented Oct 7, 2024

haeniya commented Oct 7, 2024 • edited Loading

sid-acryl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haeniya Oct 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haeniya commented Oct 7, 2024 • edited Loading

hsheth2 commented Oct 8, 2024

haeniya commented Oct 14, 2024

hsheth2 commented Oct 18, 2024

hsheth2 commented Oct 23, 2024

coderabbitai bot commented Aug 13, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

haeniya commented Sep 16, 2024 •

edited

Loading

haeniya commented Oct 7, 2024 •

edited

Loading

haeniya Oct 7, 2024 •

edited

Loading

haeniya commented Oct 7, 2024 •

edited

Loading