more type-parsing fixes #28100

edgao · 2023-07-10T19:59:35Z

see unit tests for example schemas. fixes https://github.com/airbytehq/oncall/issues/2463

edgao · 2023-07-10T20:02:09Z

.../src/test/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteTypeTest.java

+
+    AirbyteType expected = new OneOf(List.of(
+        AirbyteProtocolType.STRING,
+        new Struct(new LinkedHashMap<>() {{


I'm being lazy, lmk if you object to this

octavia-squidington-iii · 2023-07-10T20:29:04Z

destination-bigquery test report (commit `5df1792d01`) - ✅

⏲️ Total pipeline duration: 26mn43s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

edgao · 2023-07-10T20:32:15Z

.../src/test/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteTypeTest.java

+        new Array(AirbyteProtocolType.STRING),
+        // This is bad behavior, but it matches current behavior so we'll test it.
+        // Ideally, we would recognize that the sub-oneOfs are also objects.
+        new OneOf(List.of(new Struct(new LinkedHashMap<>()))),


I guess technically this is also currently impossible behavior, because there's no way to generate nested OneOfs (only nested UnsupportedOneOf). 🤷

octavia-squidington-iii · 2023-07-10T20:46:07Z

destination-bigquery-denormalized test report (commit `5df1792d01`) - ✅

⏲️ Total pipeline duration: 16mn54s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

octavia-squidington-iii · 2023-07-10T21:11:54Z

destination-bigquery-denormalized test report (commit `15111a5c1f`) - ✅

⏲️ Total pipeline duration: 18mn37s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

cynthiaxyin

Thanks for fixing this case!

cynthiaxyin · 2023-07-10T21:22:29Z

...ping/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteType.java

@@ -168,12 +176,16 @@ record OneOf(List<AirbyteType> options) implements AirbyteType {
    public LinkedHashMap<String, AirbyteType> asColumns() {
      final long numObjectOptions = options.stream().filter(o -> o instanceof Struct).count();
      if (numObjectOptions > 1) {
-        throw new IllegalArgumentException("Can't extract columns from a schema with multiple object options");
+        LOGGER.warn("Found multiple object options in a oneOf schema. This is not supported.");
+        return new LinkedHashMap<>();


Alternatively, we could pull the columns from the first struct? :P but maybe that's more confusing than helpful

this situation isn't actually possible right now, so 🤷 we can answer that question whenever we merge UnsupportedOneOf and OneOf

...ping/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/AirbyteType.java

octavia-squidington-iii · 2023-07-10T21:37:10Z

destination-bigquery test report (commit `15111a5c1f`) - ✅

⏲️ Total pipeline duration: 25mn07s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-07-10T22:04:55Z

destination-bigquery test report (commit `1bf7de87a7`) - ✅

⏲️ Total pipeline duration: 25mn05s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-07-10T22:21:50Z

destination-bigquery-denormalized test report (commit `1bf7de87a7`) - ✅

⏲️ Total pipeline duration: 16mn46s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)"" This reverts commit 348c577. * version bumps+changelog * Speed up BQ by having 2 queries, and not an OR (#27981) * 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856) * only run t+d code if it's enabled * dockerfile+changelog * remove changelog entry * Destinations V2: handle optional fields for `object` and `array` types (#27898) * catch null schema * fix null properties * clean up * consolidate + add more tests * try catch * empty json test * Automated Commit - Formatting Changes * remove todo * destination bigquery: misc updates to 1s1t code (#28057) * switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build * more type-parsing fixes (#28100) * more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <[email protected]> * Automated Commit - Formatting Changes * Improve protocol type parsing (#28126) * Automated Commit - Formatting Changes * Change from T&D every 10k records to an increasing time based interval (#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <[email protected]> * Simplify and speed up CDC delete support [DestinationsV2] (#28029) * Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <[email protected]> * update method name * Automated Commit - Formatting Changes * depend on pinned normalization version * implement 1s1t DATs for destination-bigquery (#27852) * intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <[email protected]> Co-authored-by: octavia-approvington <[email protected]> * Destinations V2: move create raw tables earlier (#28255) * move create raw tables * better log message * stop building normalization (#28256) * fix ability to run tests * disable incremental t+d for now * Automated Commit - Formatting Changes --------- Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Cynthia Yin <[email protected]> Co-authored-by: cynthiaxyin <[email protected]> Co-authored-by: edgao <[email protected]> Co-authored-by: Joe Bell <[email protected]> Co-authored-by: jbfbell <[email protected]> Co-authored-by: octavia-approvington <[email protected]>

* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (airbytehq#27268)"" This reverts commit 348c577. * version bumps+changelog * Speed up BQ by having 2 queries, and not an OR (airbytehq#27981) * 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (airbytehq#27856) * only run t+d code if it's enabled * dockerfile+changelog * remove changelog entry * Destinations V2: handle optional fields for `object` and `array` types (airbytehq#27898) * catch null schema * fix null properties * clean up * consolidate + add more tests * try catch * empty json test * Automated Commit - Formatting Changes * remove todo * destination bigquery: misc updates to 1s1t code (airbytehq#28057) * switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build * more type-parsing fixes (airbytehq#28100) * more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <[email protected]> * Automated Commit - Formatting Changes * Improve protocol type parsing (airbytehq#28126) * Automated Commit - Formatting Changes * Change from T&D every 10k records to an increasing time based interval (airbytehq#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <[email protected]> * Simplify and speed up CDC delete support [DestinationsV2] (airbytehq#28029) * Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (airbytehq#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <[email protected]> * update method name * Automated Commit - Formatting Changes * depend on pinned normalization version * implement 1s1t DATs for destination-bigquery (airbytehq#27852) * intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <[email protected]> Co-authored-by: octavia-approvington <[email protected]> * Destinations V2: move create raw tables earlier (airbytehq#28255) * move create raw tables * better log message * stop building normalization (airbytehq#28256) * fix ability to run tests * disable incremental t+d for now * Automated Commit - Formatting Changes --------- Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Cynthia Yin <[email protected]> Co-authored-by: cynthiaxyin <[email protected]> Co-authored-by: edgao <[email protected]> Co-authored-by: Joe Bell <[email protected]> Co-authored-by: jbfbell <[email protected]> Co-authored-by: octavia-approvington <[email protected]>

edgao added 2 commits July 10, 2023 12:53

more type-parsing fixes

0aa620b

handle duplicates

5df1792

edgao requested a review from cynthiaxyin July 10, 2023 19:59

edgao commented Jul 10, 2023

View reviewed changes

edgao and others added 2 commits July 10, 2023 20:20

Automated Commit - Format and Process Resources Changes

f419016

add tests for asColumns

898650f

edgao commented Jul 10, 2023

View reviewed changes

Automated Commit - Format and Process Resources Changes

15111a5

edgao added 2 commits July 10, 2023 14:17

log warnings instead of throwing exception

15744c9

better log message

416474f

cynthiaxyin approved these changes Jul 10, 2023

View reviewed changes

error level

4f41ea9

edgao merged commit 57422c1 into edgao/1s1t_redeploy Jul 10, 2023

edgao deleted the edgao/oneof_object branch July 10, 2023 21:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

more type-parsing fixes #28100

more type-parsing fixes #28100

edgao commented Jul 10, 2023

edgao Jul 10, 2023

octavia-squidington-iii commented Jul 10, 2023

edgao Jul 10, 2023

octavia-squidington-iii commented Jul 10, 2023

octavia-squidington-iii commented Jul 10, 2023

cynthiaxyin left a comment

cynthiaxyin Jul 10, 2023

edgao Jul 10, 2023

octavia-squidington-iii commented Jul 10, 2023

octavia-squidington-iii commented Jul 10, 2023

octavia-squidington-iii commented Jul 10, 2023

more type-parsing fixes #28100

more type-parsing fixes #28100

Conversation

edgao commented Jul 10, 2023

edgao Jul 10, 2023

Choose a reason for hiding this comment

octavia-squidington-iii commented Jul 10, 2023

destination-bigquery test report (commit 5df1792d01) - ✅

edgao Jul 10, 2023

Choose a reason for hiding this comment

octavia-squidington-iii commented Jul 10, 2023

destination-bigquery-denormalized test report (commit 5df1792d01) - ✅

octavia-squidington-iii commented Jul 10, 2023

destination-bigquery-denormalized test report (commit 15111a5c1f) - ✅

cynthiaxyin left a comment

Choose a reason for hiding this comment

cynthiaxyin Jul 10, 2023

Choose a reason for hiding this comment

edgao Jul 10, 2023

Choose a reason for hiding this comment

octavia-squidington-iii commented Jul 10, 2023

destination-bigquery test report (commit 15111a5c1f) - ✅

octavia-squidington-iii commented Jul 10, 2023

destination-bigquery test report (commit 1bf7de87a7) - ✅

octavia-squidington-iii commented Jul 10, 2023

destination-bigquery-denormalized test report (commit 1bf7de87a7) - ✅

destination-bigquery test report (commit `5df1792d01`) - ✅

destination-bigquery-denormalized test report (commit `5df1792d01`) - ✅

destination-bigquery-denormalized test report (commit `15111a5c1f`) - ✅

destination-bigquery test report (commit `15111a5c1f`) - ✅

destination-bigquery test report (commit `1bf7de87a7`) - ✅

destination-bigquery-denormalized test report (commit `1bf7de87a7`) - ✅