Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix glue_table parameter not filtering tables #335

Merged
merged 2 commits into from
Jul 17, 2024
Merged

Fix glue_table parameter not filtering tables #335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
414b31b
Fix glue_table parameter not filtering tables issue
roykoand Jul 17, 2024
d4ca5a4
Fix typo in glue test
roykoand Jul 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added
- Add support for dbt manifest file (#104)
- Fix import of pyspark for type-checking when pyspark isn't required as a module (#312)- `datacontract import --format spark`: Import from Spark tables (#326)
- Fix import of pyspark for type-checking when pyspark isn't required as a module (#312)- `datacontract import --format spark`: Import from Spark tables (#326)
- Fix an issue where specifying `glue_table` as parameter did not filter the tables and returned all tables instead from `source` instead (#333)

## [0.10.9] - 2024-07-03

Expand Down
4 changes: 2 additions & 2 deletions datacontract/imports/glue_importer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ class GlueImporter(Importer):
def import_source(
self, data_contract_specification: DataContractSpecification, source: str, import_args: dict
) -> dict:
return import_glue(data_contract_specification, source, import_args.get("glue_tables"))
return import_glue(data_contract_specification, source, import_args.get("glue_table"))


def get_glue_database(database_name: str):
Expand Down Expand Up @@ -154,7 +154,7 @@ def import_glue(
for column in table_schema:
field = create_typed_field(column["Type"])

# hive partitons are required, but are not primary keys
# hive partitions are required, but are not primary keys
if column.get("Hive"):
field.required = True

Expand Down
2 changes: 1 addition & 1 deletion tests/test_import_glue.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ def test_import_glue_schema(setup_mock_glue):

@mock_aws
def test_import_glue_schema_with_table_filters(setup_mock_glue):
result = DataContract().import_from_source(format="glue", source="test_database", glue_tables=["table_1"])
result = DataContract().import_from_source(format="glue", source="test_database", glue_table=["table_1"])

# we specify a table that the Mock doesn't have and thus expect an empty result
with open("fixtures/glue/datacontract-empty-model.yaml") as file:
Expand Down