-
Notifications
You must be signed in to change notification settings - Fork 4.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Destination bigquery: rerelease 1s1t behind gate (#27936)
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)"" This reverts commit 348c577. * version bumps+changelog * Speed up BQ by having 2 queries, and not an OR (#27981) * 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856) * only run t+d code if it's enabled * dockerfile+changelog * remove changelog entry * Destinations V2: handle optional fields for `object` and `array` types (#27898) * catch null schema * fix null properties * clean up * consolidate + add more tests * try catch * empty json test * Automated Commit - Formatting Changes * remove todo * destination bigquery: misc updates to 1s1t code (#28057) * switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build * more type-parsing fixes (#28100) * more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <[email protected]> * Automated Commit - Formatting Changes * Improve protocol type parsing (#28126) * Automated Commit - Formatting Changes * Change from T&D every 10k records to an increasing time based interval (#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <[email protected]> * Simplify and speed up CDC delete support [DestinationsV2] (#28029) * Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <[email protected]> * update method name * Automated Commit - Formatting Changes * depend on pinned normalization version * implement 1s1t DATs for destination-bigquery (#27852) * intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <[email protected]> Co-authored-by: octavia-approvington <[email protected]> * Destinations V2: move create raw tables earlier (#28255) * move create raw tables * better log message * stop building normalization (#28256) * fix ability to run tests * disable incremental t+d for now * Automated Commit - Formatting Changes --------- Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Cynthia Yin <[email protected]> Co-authored-by: cynthiaxyin <[email protected]> Co-authored-by: edgao <[email protected]> Co-authored-by: Joe Bell <[email protected]> Co-authored-by: jbfbell <[email protected]> Co-authored-by: octavia-approvington <[email protected]>
- Loading branch information
8 people
authored
Jul 14, 2023
1 parent
0d185a2
commit 934acaa
Showing
83 changed files
with
5,031 additions
and
130 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
75 changes: 75 additions & 0 deletions
75
...rations/bases/base-java/src/main/java/io/airbyte/integrations/base/DestinationConfig.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| /* | ||
| * Copyright (c) 2023 Airbyte, Inc., all rights reserved. | ||
| */ | ||
|
|
||
| package io.airbyte.integrations.base; | ||
|
|
||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import com.google.common.annotations.VisibleForTesting; | ||
| import jakarta.inject.Singleton; | ||
| import org.slf4j.Logger; | ||
| import org.slf4j.LoggerFactory; | ||
|
|
||
| /** | ||
| * Singleton of destination config for easy lookup of values. | ||
| */ | ||
| @Singleton | ||
| public class DestinationConfig { | ||
|
|
||
| private static final Logger LOGGER = LoggerFactory.getLogger(DestinationConfig.class); | ||
|
|
||
| private static DestinationConfig config; | ||
|
|
||
| @VisibleForTesting | ||
| protected JsonNode root; | ||
|
|
||
| private DestinationConfig() {} | ||
|
|
||
| public static void initialize(final JsonNode root) { | ||
| if (config == null) { | ||
| if (root == null) { | ||
| throw new IllegalArgumentException("Cannot create DestinationConfig from null."); | ||
| } | ||
| config = new DestinationConfig(); | ||
| config.root = root; | ||
| } else { | ||
| LOGGER.warn("Singleton was already initialized."); | ||
| } | ||
| } | ||
|
|
||
| public static DestinationConfig getInstance() { | ||
| if (config == null) { | ||
| throw new IllegalStateException("Singleton not initialized."); | ||
| } | ||
| return config; | ||
| } | ||
|
|
||
| public JsonNode getNodeValue(final String key) { | ||
| final JsonNode node = config.root.get(key); | ||
| if (node == null) { | ||
| LOGGER.debug("Cannot find node with key {} ", key); | ||
| } | ||
| return node; | ||
| } | ||
|
|
||
| // string value, otherwise empty string | ||
| public String getTextValue(final String key) { | ||
| final JsonNode node = getNodeValue(key); | ||
| if (node == null || !node.isTextual()) { | ||
| LOGGER.debug("Cannot retrieve text value for node with key {}", key); | ||
| return ""; | ||
| } | ||
| return node.asText(); | ||
| } | ||
|
|
||
| // boolean value, otherwise false | ||
| public Boolean getBooleanValue(final String key) { | ||
| final JsonNode node = getNodeValue(key); | ||
| if (node == null || !node.isBoolean()) { | ||
| LOGGER.debug("Cannot retrieve boolean value for node with key {}", key); | ||
| return false; | ||
| } | ||
| return node.asBoolean(); | ||
| } | ||
|
|
||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
13 changes: 13 additions & 0 deletions
13
...ons/bases/base-java/src/main/java/io/airbyte/integrations/base/TypingAndDedupingFlag.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| /* | ||
| * Copyright (c) 2023 Airbyte, Inc., all rights reserved. | ||
| */ | ||
|
|
||
| package io.airbyte.integrations.base; | ||
|
|
||
| public class TypingAndDedupingFlag { | ||
|
|
||
| public static final boolean isDestinationV2() { | ||
| return DestinationConfig.getInstance().getBooleanValue("use_1s1t_format"); | ||
| } | ||
|
|
||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
65 changes: 65 additions & 0 deletions
65
...ons/bases/base-java/src/test/java/io/airbyte/integrations/base/DestinationConfigTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| /* | ||
| * Copyright (c) 2023 Airbyte, Inc., all rights reserved. | ||
| */ | ||
|
|
||
| package io.airbyte.integrations.base; | ||
|
|
||
| import static org.junit.jupiter.api.Assertions.assertEquals; | ||
| import static org.junit.jupiter.api.Assertions.assertFalse; | ||
| import static org.junit.jupiter.api.Assertions.assertNotNull; | ||
| import static org.junit.jupiter.api.Assertions.assertNull; | ||
| import static org.junit.jupiter.api.Assertions.assertThrows; | ||
| import static org.junit.jupiter.api.Assertions.assertTrue; | ||
|
|
||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import io.airbyte.commons.json.Jsons; | ||
| import org.junit.jupiter.api.Test; | ||
|
|
||
| public class DestinationConfigTest { | ||
|
|
||
| private static final String JSON = """ | ||
| { | ||
| "foo": "bar", | ||
| "baz": true | ||
| } | ||
| """; | ||
|
|
||
| private static final JsonNode NODE = Jsons.deserialize(JSON); | ||
|
|
||
| @Test | ||
| public void testInitialization() { | ||
| // bad initialization | ||
| assertThrows(IllegalArgumentException.class, () -> DestinationConfig.initialize(null)); | ||
| assertThrows(IllegalStateException.class, DestinationConfig::getInstance); | ||
|
|
||
| // good initialization | ||
| DestinationConfig.initialize(NODE); | ||
| assertNotNull(DestinationConfig.getInstance()); | ||
| assertEquals(NODE, DestinationConfig.getInstance().root); | ||
|
|
||
| // initializing again doesn't change the config | ||
| final JsonNode nodeUnused = Jsons.deserialize("{}"); | ||
| DestinationConfig.initialize(nodeUnused); | ||
| assertEquals(NODE, DestinationConfig.getInstance().root); | ||
| } | ||
|
|
||
| @Test | ||
| public void testValues() { | ||
| DestinationConfig.initialize(NODE); | ||
|
|
||
| assertEquals("bar", DestinationConfig.getInstance().getTextValue("foo")); | ||
| assertEquals("", DestinationConfig.getInstance().getTextValue("baz")); | ||
|
|
||
| assertFalse(DestinationConfig.getInstance().getBooleanValue("foo")); | ||
| assertTrue(DestinationConfig.getInstance().getBooleanValue("baz")); | ||
|
|
||
| // non-existent key | ||
| assertEquals("", DestinationConfig.getInstance().getTextValue("blah")); | ||
| assertFalse(DestinationConfig.getInstance().getBooleanValue("blah")); | ||
|
|
||
| assertEquals(Jsons.deserialize("\"bar\""), DestinationConfig.getInstance().getNodeValue("foo")); | ||
| assertEquals(Jsons.deserialize("true"), DestinationConfig.getInstance().getNodeValue("baz")); | ||
| assertNull(DestinationConfig.getInstance().getNodeValue("blah")); | ||
| } | ||
|
|
||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.