Follow "More flexible cluster configuration".

[ueshin]

Description

Follows "More flexible cluster configuration" at dbt-labs/dbt-spark#467.

• Reuse dbt-spark's implementation
• Remove the dependency on databricks-cli
• Internal refactorings

[allisonwang-db]

This looks good! Thanks for the refactoring.

[allisonwang-db]

Can we specify an http_path that is different from the one in dbt profiles for a Python model?

[ueshin]

Thanks for the heads-up.
Actually I double-checked it and found there is a mistake. I'll update it.

After the update, yes, we can specify http_path in the Python model config that is different from the one in the profile.

def model(...):
    dbt.config(http_path='...')
    ....

or in dbt_project.yml file:

models:
  - name: my_python_model
    config:
      http_path: ...

[ueshin]

Thanks! merging.