Supports SQL in DatabricksHook

[mik-laj]

Description

Currently, we support submitting spark jobs for the Databricks service. On December 15, 2021, they announced General Availability of Databricks SQL and we should provide support for it also.

To do this, we can use one of the existing libraries that provide integration with SQLAlchemy/DBApi:

• https://pypi.org/project/databricks-sql-connector/ - Apache 2 Licensed (?), officially maintained by Databricks, code source is not available publicly (yet?)
• https://github.com/crflynn/databricks-dbapi - MIT Licensed, unmaintained, open-source public available

In particular, we should consider using DbApiHook class.

CC: @eskarimov @alexott @christophergrant @Mokubyow

Use case/motivation

Support a new kind of task.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[alexott]

It’s basically #21030 that I’ve filed today

[mik-laj]

Ahh yes. So I close this PR. I opened this PR because I saw that Apache Superset has a new issue related to Databricks.
apache/superset#18139