Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use AttrDict instead of agate #604

Merged
merged 11 commits into from
Jan 26, 2023
Merged

Use AttrDict instead of agate #604

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
268a22d
add mypy ignore to column, connections and init
colin-rogers-dbt Sep 13, 2022
317c163
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Sep 26, 2022
3a55d62
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Sep 27, 2022
f0f9312
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Sep 30, 2022
bb4d7d0
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Dec 9, 2022
a1e0425
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Dec 13, 2022
9fb48c5
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Dec 20, 2022
051c2e5
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Dec 23, 2022
4637e7e
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Jan 11, 2023
7ae7dc0
Merge branch 'main' of https://github.com/dbt-labs/dbt-spark
colin-rogers-dbt Jan 20, 2023
54d34a6
change from agate to AttrDict
colin-rogers-dbt Jan 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions dbt/adapters/spark/impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from dbt.adapters.base import BaseRelation
from dbt.clients.agate_helper import DEFAULT_TYPE_TESTER
from dbt.events import AdapterLogger
from dbt.utils import executor
from dbt.utils import executor, AttrDict

logger = AdapterLogger("Spark")

Expand Down Expand Up @@ -176,9 +176,7 @@ def get_relation(self, database: str, schema: str, identifier: str) -> Optional[

return super().get_relation(database, schema, identifier)

def parse_describe_extended(
self, relation: Relation, raw_rows: List[agate.Row]
) -> List[SparkColumn]:
def parse_describe_extended(self, relation: Relation, raw_rows: AttrDict) -> List[SparkColumn]:
# Convert the Row to a dict
dict_rows = [dict(zip(row._keys, row._values)) for row in raw_rows]
# Find the separator between the rows and the metadata provided
Expand Down Expand Up @@ -218,7 +216,7 @@ def find_table_information_separator(rows: List[dict]) -> int:
def get_columns_in_relation(self, relation: Relation) -> List[SparkColumn]:
columns = []
try:
rows: List[agate.Row] = self.execute_macro(
rows: AttrDict = self.execute_macro(
GET_COLUMNS_IN_RELATION_RAW_MACRO_NAME, kwargs={"relation": relation}
)
columns = self.parse_describe_extended(relation, rows)
Expand Down