Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support filter_schema setting #215

Closed
ShahBinoy opened this issue Aug 23, 2023 · 2 comments · Fixed by #218
Closed

Support filter_schema setting #215

ShahBinoy opened this issue Aug 23, 2023 · 2 comments · Fixed by #218

Comments

@ShahBinoy
Copy link

I am trying to perform ELT with meltano elt command against Postgres Tap (MeltanoLabs)
But I am getting error

psycopg2.errors.InsufficientPrivilege: permission denied for schema pdr_ops

Since the remote database is a third party Database, we do not have access to pdr_ops schema. Why is it trying to access pdr_ops schema. I only need to check against pdr schema

How can I limit my operations to only one schema pdr and not face this permissions error ?

meltano.yaml

  - name: pdr
    inherit_from: tap-postgres
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git
    capabilities: 
    - state
    select:
    - pdr-mv_participant_all.participant_id
    - pdr-mv_participant_all.id
    - pdr-mv_participant_all.modified
    metadata:
      pdr-mv_participant_all:
        replication-method: INCREMENTAL
        replication-key: modified
        view-key-properties: [id]
@visch
Copy link
Member

visch commented Aug 23, 2023

https://github.com/meltano/sdk/blob/main/singer_sdk/connectors/sql.py#L371 is a good place to look I think we could add a config here. Maybe there's better options here though.

@visch
Copy link
Member

visch commented Aug 23, 2023

meltano/sdk#1234 @edgarrmondragon pointing to as another issue we may have to add a feature for as well. We'll want to add a test here on the database as well which will take a bit more time!

visch pushed a commit that referenced this issue Aug 25, 2023
Makes use of config filter_schemas to only pull specified schemas,
increasing performance and allowing users to sidestep permissions issues
for schemas they don't have access to.

Closes #215
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants