Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IAM docs #949

Merged
merged 1 commit into from
Jan 3, 2025
Merged

Add IAM docs #949

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ This tool is runnable locally, or on any other backend supported by Apache Beam,

## Overview

DBeam is tool based that reads all the data from single SQL database table,
DBeam is a tool that reads all the data from single SQL database table,
converts the data into [Avro](https://avro.apache.org/) and stores it into
appointed location, usually in GCS.
It runs as a single threaded [Apache Beam](https://beam.apache.org/) pipeline.
Expand All @@ -33,7 +33,7 @@ simply streams the table contents via JDBC into target location as Avro.

- Supports both PostgreSQL and MySQL JDBC connectors
- Supports [Google CloudSQL](https://cloud.google.com/sql/) managed databases
- Currently output only to Avro format
- Currently outputs only to Avro format
- Reads database from an external password file (`--passwordFile`) or an external [KMS](https://cloud.google.com/kms/) encrypted password file (`--passwordFileKmsEncrypted`)
- Can filter only records of the current day with the `--partitionColumn` parameter
- Check and fail on too old partition dates. Snapshot dumps are not filtered by a given date/partition, when running for a too old partition, the job fails to avoid new data in old partitions. (can be disabled with `--skipPartitionCheck`)
Expand Down Expand Up @@ -237,6 +237,10 @@ java \
...
```

### IAM authentication

When using Google Cloud, [IAM authentication](https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory/blob/main/docs/jdbc.md#iam-authentication) may be used by adding `enableIamAuth=true` to the JDBC URL, providing a dummy password, and following the appropriate rules for user naming with respect to the Service Account name.

## Using as a library


Expand Down
Loading