Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADAP-1167: Add table format telemetry reporting to Athena adapter #403

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6948977
added table format telemetry reporting to athena adapter
VanTudor Jan 13, 2025
4d77a53
linting + code quality
VanTudor Jan 13, 2025
f3a23f4
removed some debugging print statements
VanTudor Jan 13, 2025
8808fee
Merge branch 'main' into ADAP-1167/add-athena-adapter-telemetry
VanTudor Jan 14, 2025
13de64e
Merge branch 'main' into ADAP-1167/add-athena-adapter-telemetry
VanTudor Jan 15, 2025
da003bd
linting
VanTudor Jan 15, 2025
36b245f
Merge branch 'main' into ADAP-1167/add-athena-adapter-telemetry
VanTudor Jan 16, 2025
592f255
added extra debug logs via pyproject.toml
VanTudor Jan 16, 2025
c1f9b89
changed iceberg retries parallelism to 2 from 10
VanTudor Jan 16, 2025
705e11c
forced some integration tests to run in series for athena, during Git…
VanTudor Jan 17, 2025
2474f38
made the is-flaky-tests flag available outside the _integration-tests…
VanTudor Jan 17, 2025
2bc8856
naming improvements to workflow , pertaining to flaky test changes
VanTudor Jan 17, 2025
c1c0d5a
fixed syntac error in _integration-tests.yml
VanTudor Jan 17, 2025
8dbdf7c
fixed hatch run tests command for athena
VanTudor Jan 17, 2025
b0309a9
removed tag isolation for iceberg retries retries tests
VanTudor Jan 17, 2025
1fb668f
added flaky tag to a test
VanTudor Jan 17, 2025
09ea593
disabled a very flaky iceberg retries disabled test
VanTudor Jan 20, 2025
ce47cba
Merge branch 'main' into ADAP-1167/add-athena-adapter-telemetry
VanTudor Jan 23, 2025
ea18acc
Cleaner implementation of _get_adapter_specific_run_info
VanTudor Jan 23, 2025
baef211
removed flaky flag from iceberg retry test
VanTudor Jan 23, 2025
8451a92
athena: rolled back pyproject.toml changes
VanTudor Jan 23, 2025
f62419e
Merge branch 'main' into ADAP-1167/add-athena-adapter-telemetry
VanTudor Jan 27, 2025
6a1b696
Merge branch 'main' into ADAP-1167/add-athena-adapter-telemetry
mikealfare Jan 27, 2025
7dc3f79
Merge branch 'main' into ADAP-1167/add-athena-adapter-telemetry
mikealfare Jan 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions dbt-athena/src/dbt/adapters/athena/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -1452,3 +1452,14 @@ def _run_query(self, sql: str, catch_partitions_limit: bool) -> AthenaCursor:
LOGGER.debug(f"CAUGHT EXCEPTION: {e}")
raise e
return cursor

@classmethod
def _get_adapter_specific_run_info(cls, config: RelationConfig) -> Dict[str, Any]:
try:
table_format = config._extra.get("table_type")
except AttributeError:
table_format = None
return {
"adapter_type": "athena",
"table_format": table_format,
}
3 changes: 3 additions & 0 deletions dbt-athena/tests/functional/adapter/test_retries_iceberg.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@
seeds__expected_target_post = "id,status\n" + "\n".join([f"{i},{i}" for i in range(PARALLELISM)])


@pytest.mark.skip(
reason="Unreliable test in practice. No apparent reliable way to trigger a retry, which causes the test to fail. Disabled until we get a resolution here https://github.com/dbt-labs/dbt-athena/pull/657/files#r1922043351"
VanTudor marked this conversation as resolved.
Show resolved Hide resolved
)
class TestIcebergRetriesDisabled:
@pytest.fixture(scope="class")
def dbt_profile_target(self):
Expand Down
28 changes: 28 additions & 0 deletions dbt-athena/tests/unit/test_adapter_telemetry.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
from unittest import mock

import dbt.adapters.athena.__version__

from dbt.adapters.athena.impl import AthenaAdapter
from dbt.adapters.base.relation import AdapterTrackingRelationInfo


def test_telemetry_with_athena_details():
mock_model_config = mock.MagicMock()
mock_model_config._extra = mock.MagicMock()
mock_model_config._extra = {
"adapter_type": "athena",
"table_type": "iceberg",
}

res = AthenaAdapter.get_adapter_run_info(mock_model_config)

assert res.adapter_name == "athena"
assert res.base_adapter_version == dbt.adapters.__about__.version
assert res.adapter_version == dbt.adapters.athena.__version__.version

assert res.model_adapter_details == {
"adapter_type": "athena",
"table_format": "iceberg",
}

assert isinstance(res, AdapterTrackingRelationInfo)