file-based CDK: Configurable strings_can_be_null #29298
Merged
+652
−513
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
girarda
commented
Aug 9, 2023
| @@ -210,13 +230,6 @@ def _cast_types( | |||
| prop_type = property_types.get(key) | |||
| cast_value: Any = value | |||
|
|
|||
| if isinstance(prop_type, list): | |||
There was a problem hiding this comment.
moved outside the function so the deduped property types can also be used in _to_nullable
girarda
commented
Aug 9, 2023
| @@ -43,6 +43,18 @@ | |||
| "description": "Used during spec; allows the developer to configure the cloud provider specific options\nthat are needed when users configure a file-based source.", | |||
| "type": "object", | |||
| "properties": { | |||
| "start_date": { | |||
There was a problem hiding this comment.
not 100% sure why this is being re-ordered, but properties is a map so this shouldn't matter
girarda
changed the title
harrytou
pushed a commit
to KYVENetwork/airbyte
that referenced
this pull request
Sep 1, 2023
* [ISSUE airbytehq#28893] infer csv schema * [ISSUE airbytehq#28893] align with pyarrow * Automated Commit - Formatting Changes * [ISSUE airbytehq#28893] legacy inference and infer only when needed * [ISSUE airbytehq#28893] fix scenario tests * [ISSUE airbytehq#28893] using discovered schema as part of read * [ISSUE airbytehq#28893] self-review + cleanup * [ISSUE airbytehq#28893] fix test * [ISSUE airbytehq#28893] code review part #1 * [ISSUE airbytehq#28893] code review part #2 * Fix test * formatcdk * first pass * [ISSUE airbytehq#28893] code review * fix mypy issues * comment * rename for clarity * Add a scenario test case * this isn't optional anymore * FIX test log level * Re-adding failing tests * [ISSUE airbytehq#28893] improve inferrence to consider multiple types per value * Automated Commit - Formatting Changes * [ISSUE airbytehq#28893] remove InferenceType.PRIMITIVE_AND_COMPLEX_TYPES * Code review * Automated Commit - Formatting Changes * fix unit tests --------- Co-authored-by: maxi297 <[email protected]> Co-authored-by: maxi297 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
How
When converting values to null:
Recommended reading order
airbyte-cdk/python/airbyte_cdk/sources/file_based/config/csv_format.pyairbyte-cdk/python/airbyte_cdk/sources/file_based/file_types/csv_parser.py