-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Destination bigquery: rerelease 1s1t behind gate #27936
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
Edit... I closed the wrong PR... |
...ping/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteType.java
Outdated
Show resolved
Hide resolved
...src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteTypeUtils.java
Show resolved
Hide resolved
...src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteTypeUtils.java
Show resolved
Hide resolved
...src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteTypeUtils.java
Show resolved
Hide resolved
...src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteTypeUtils.java
Show resolved
Hide resolved
...src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteTypeUtils.java
Show resolved
Hide resolved
...src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteTypeUtils.java
Show resolved
Hide resolved
...ava/io/airbyte/integrations/destination/buffered_stream_consumer/BufferedStreamConsumer.java
Outdated
Show resolved
Hide resolved
} | ||
|
||
public ParsedCatalog parseCatalog(ConfiguredAirbyteCatalog catalog) { | ||
// this code is bad and I feel bad |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❤️
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will create a ticket for fixing this, seems like too much of a mess to improve/untangle right now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, this only affects 1s1t output anyway.
we also have #27798, which probably touches this part of the codebase - anything we do here should take that into account
@@ -52,7 +51,7 @@ public static BufferCreateFunction createBufferFunction(final S3AvroFormatConfig | |||
return (pair, catalog) -> { | |||
final AirbyteStream stream = catalog.getStreams() | |||
.stream() | |||
.filter(s -> s.getStream().getName().equals(pair.getName()) && StringUtils.equals(s.getStream().getNamespace(), pair.getNamespace())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it true that even in the async world, we shouldn't have more than one namespace? Like even for CDC syncs, its multiple streams/table which are all being written to the same namespace? Does this logic have to consider any of the name mangling logic in the catalog parser?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
git says you might be the most qualified to answer that 😅 db28b13#diff-d68a5be057956e8fb021561f4c7111aa1e72eac6b87906eee8d7058a99f87163
but we explicitly support connections with two streams that have the same name but different namespace (iirc there's a DAT that verifies this...)
name mangling
depends on how exactly we're calling into this class, but StreamId has originalNamespace
/ originalName
for this exact reason - as long as we're only using the mangled names inside the sql generator, it should be fine. (I haven't verified that we're doing it correctly though)
final TimePartitioning partitioning = TimePartitioning.newBuilder(TimePartitioning.Type.DAY) | ||
.setField(JavaBaseConstants.COLUMN_NAME_EMITTED_AT) | ||
.setField(chunkingColumn) | ||
.build(); | ||
|
||
final Clustering clustering = Clustering.newBuilder() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@evantahler this is where the clustering logic is, its an array so I wonder if clustering on all columns is crazy? or maybe just leaving off the data column?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alex-gron maybe you have some advice? I don't think we should be clustering on more than one column in BigQuery perhaps? Either way, I think this is equivalent to what we are doing now - clustering by one column, COLUMN_NAME_EMITTED_AT, so I'm 👍 with it.
![Screenshot 2023-07-05 at 3 10 22 PM](https://private-user-images.githubusercontent.com/303226/251294056-bf9a418a-144b-4169-a844-9362d241ccd6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkxOTE4OTgsIm5iZiI6MTczOTE5MTU5OCwicGF0aCI6Ii8zMDMyMjYvMjUxMjk0MDU2LWJmOWE0MThhLTE0NGItNDE2OS1hODQ0LTkzNjJkMjQxY2NkNi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMjEwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDIxMFQxMjQ2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hNDQ5MzE4OWM0NzZiYWFkOGY5MzljZWZkNjVlMTA3M2M4MWMyOTUyMGQyYTQyMjY0ZWUxYjE2Nzc4Zjg2NWNiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.o0ZcTgD5mbeZYibt1ESWhH2aSPwBmms5UnQcuVIHfQE)
canonicalized = "_" + canonicalized; | ||
} | ||
|
||
// TODO this is probably wrong |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
... 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah... I cut a lot of corners back when I thought we didn't need to support name mangling in 1s1t >.>
we have a ticket to make sure 1s1t does this stuff correctly, right?
...c/main/java/io/airbyte/integrations/destination/bigquery/BigQueryStagingConsumerFactory.java
Outdated
Show resolved
Hide resolved
.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java
Outdated
Show resolved
Hide resolved
} | ||
|
||
@Override | ||
public String createTable(final StreamConfig stream, final String suffix) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this and BigQueryUtils
is where the current logic lives? But its using the Bigquery java sdk rather than SQL
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry realizing this comment isn't clear - we had talked about colocating some of the common operations. Not necessarily right now, but maybe its a good idea colocate any "create table/namespace" logic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mmhm. I don't have a strong opinion on how to make this happen - a few weeks back I was pretty dogmatic about (1) use sql for everything, (2) keep separate classes for "generate SQL" and "actually interact with the db", but I feel less strongly after actually seeing the code written
e.g. maybe we just merge sqlgenerator+destinationhandler? or kill off destinationhandler and add everything into some other existing class
.../main/java/io/airbyte/integrations/destination/bigquery/uploader/BigQueryDirectUploader.java
Outdated
Show resolved
Hide resolved
* switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build
* more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <[email protected]>
#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <[email protected]>
* Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <[email protected]>
* intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <[email protected]> Co-authored-by: octavia-approvington <[email protected]>
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-postgres-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-postgres-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-postgres-strict-encrypt:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-postgres-strict-encrypt test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-snowflake docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-snowflake:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
* move create raw tables * better log message
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-mariadb-columnstore/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-mariadb-columnstore docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-mariadb-columnstore:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mariadb-columnstore test
cancelled the PR check run; running connector tests in https://github.com/airbytehq/airbyte/actions/runs/5548043438/jobs/10130507088 ( only testing a subset of connectors so that they don't take literally forever. |
bigquery, bigquery-denormalized, redshift, gcs, and s3 are all passing tests on CI. Snowflake tests all failed due to connector crash on CI. I couldn't repro that locally; spot-checked a few tests and they all passed. We're not publishing snowflake from this branch anyway. postgres/strict-encrypt are failing... but I think that's fine. We're not publishing them, and can deal with it later. Will /approve-and-merge this pr first thing tomorrow. |
/legacy-test connector=connectors/destination-snowflake
Build PassedTest summary info:
|
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-starburst-galaxy/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-starburst-galaxy docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-starburst-galaxy:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-starburst-galaxy test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-oracle-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-oracle:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-oracle-strict-encrypt test
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-mssql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-mssql:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-mssql-strict-encrypt:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql-strict-encrypt test
/approve-and-merge reason="tested the relevant subset of connectors (#27936 (comment)) + subsequent legacy-test" |
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (airbytehq#27268)"" This reverts commit 348c577. * version bumps+changelog * Speed up BQ by having 2 queries, and not an OR (airbytehq#27981) * 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (airbytehq#27856) * only run t+d code if it's enabled * dockerfile+changelog * remove changelog entry * Destinations V2: handle optional fields for `object` and `array` types (airbytehq#27898) * catch null schema * fix null properties * clean up * consolidate + add more tests * try catch * empty json test * Automated Commit - Formatting Changes * remove todo * destination bigquery: misc updates to 1s1t code (airbytehq#28057) * switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build * more type-parsing fixes (airbytehq#28100) * more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <[email protected]> * Automated Commit - Formatting Changes * Improve protocol type parsing (airbytehq#28126) * Automated Commit - Formatting Changes * Change from T&D every 10k records to an increasing time based interval (airbytehq#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <[email protected]> * Simplify and speed up CDC delete support [DestinationsV2] (airbytehq#28029) * Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (airbytehq#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <[email protected]> * update method name * Automated Commit - Formatting Changes * depend on pinned normalization version * implement 1s1t DATs for destination-bigquery (airbytehq#27852) * intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <[email protected]> Co-authored-by: octavia-approvington <[email protected]> * Destinations V2: move create raw tables earlier (airbytehq#28255) * move create raw tables * better log message * stop building normalization (airbytehq#28256) * fix ability to run tests * disable incremental t+d for now * Automated Commit - Formatting Changes --------- Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Cynthia Yin <[email protected]> Co-authored-by: cynthiaxyin <[email protected]> Co-authored-by: edgao <[email protected]> Co-authored-by: Joe Bell <[email protected]> Co-authored-by: jbfbell <[email protected]> Co-authored-by: octavia-approvington <[email protected]>
readers: please see individual commits.
plan: