Deprecated connector for Databricks in use

[ad-m]

Is your feature request related to a problem? Please describe.

We use databricks-dbapi[sqlalchemy] since #13682 by @betodealmeida. That library is deprecated, so we should avoid using it.

On databricks-dbapi following note are available:

This package is no longer maintained. Databricks now officially maintains a DBAPI package called databricks-sql-connector that is compatible with workspace and sql analytics clusters. There is also the newer sqlalchemy-databricks package which uses the databricks-sql-connector as a driver.

Describe the solution you'd like

We should migrate to a maintained driver for Databricks. We need to choose a solution that is stable, compatible, maintained, and compatible with our project's licenses.

Describe alternatives you've considered

We could drop Databricks support, but this is not a viable alternative.

We could write our own connector, but it takes disproportionate resources.

Additional context

sqlalchemy-databricks package is MIT licensed package.
databricks-sql-connector package is currently shared in stage "Public Preview" as Apache 2.0 licensed. As such, it raises the possibility of its incorporation at this stage into the public release of Apache Superset. As such, it raises doubts about its applicability at this stage to the public release of Apache Superset.