[Bug] Issue with codegen, requires changing dbt/adapters/databricks/relation.py

[kalebkemp]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Let me start by saying, I am aware this is a codegen command, but I was able to fix it by changing code for the databricks dbt adapter in my local python virtual environment. If I need to log an issue in the codegen repo* instead, please let me know..

Get an error when running generate_source command.

18:48:54  Encountered an error while running operation: Database Error
  Field "type" of type Optional[DatabricksRelationType] in DatabricksRelation has invalid value 'managed_shallow_clone'

Expected Behavior

Codegen is able to connect to databricks and generate a proper source file

Steps To Reproduce

Using databricks adapter,
Create a source from shallow cloned tables
In databricks, create new schema, and create tables by:

CREATE TABLE my_dev_catalog.my_schema.my_table shallow clone my_prod_catalog.my_schema.my_table

Run this dbt command:

dbt run-operation generate_source --args '{"schema_name": "my_schema", "database_name": "my_dev_catalog" }' > models/staging/my_dev_schema/_my_dev_schema_sources.yml

Relevant log output

18:48:54  Encountered an error while running operation: Database Error
  Field "type" of type Optional[DatabricksRelationType] in DatabricksRelation has invalid value 'managed_shallow_clone'

Environment

- OS: Windows 11
- Python: 3.12.4
- dbt: 1.8.8

Which database adapter are you using with dbt?

No response

Additional Context

Issue was fixed locally for me by changing .venv/Libs/site-packages/dbt/adapters/databricks/relation.py to have "managed_shallow_clone" as a databricks relation type (ShallowClone line below)

class DatabricksRelationType(StrEnum):
    Table = "table"
    View = "view"
    CTE = "cte"
    MaterializedView = "materialized_view"
    Foreign = "foreign"
    StreamingTable = "streaming_table"
    ShallowClone = "managed_shallow_clone" # ADDED THIS LINE
    External = "external"
    Unknown = "unknown"

Though I am not sure the implications of just making this change. It has not caused any issues locally for me in months. I just have to remember to do this again if I ever clone my repo to a new folder or machine and create a new venv.

[kalebkemp]

I realize I perhaps should have logged this under dbt-databricks.... let me know the proper place..

[dbeatty10]

@kalebkemp Could you open an issue in dbt-databricks and share the link here once you've done that? I'm going to keep this issue open for the meantime, but transferring it to dbt-codegen.

[dbeatty10]

Closing in favor of databricks/dbt-databricks#863 databricks/dbt-databricks#873