-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug: fixing async syntax for python 3.7 #5759
Bug: fixing async syntax for python 3.7 #5759
Conversation
Rename async to async_ so superset installs for python 3.7.
Codecov Report
@@ Coverage Diff @@
## master #5759 +/- ##
==========================================
+ Coverage 63.34% 63.36% +0.02%
==========================================
Files 364 364
Lines 23051 23051
Branches 2565 2565
==========================================
+ Hits 14602 14607 +5
+ Misses 8434 8429 -5
Partials 15 15
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This turned out to be more complicated than I expected! I'll make a quick PR to Pyhive to fix their side.
tests/celery_tests.py
Outdated
self.login() | ||
resp = self.client.post( | ||
'/superset/sql_json/', | ||
data=dict( | ||
database_id=db_id, | ||
sql=sql, | ||
async=async, | ||
async_=async_, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here we're POST
ing the data payload into the sql_json
view, so this should actually be runAsync=async_
. See: https://github.com/apache/incubator-superset/blob/60ecd72aac85b8faddeeb339f24299224ecbca91/superset/views/core.py#L2375
superset/db_engine_specs.py
Outdated
@@ -370,7 +370,7 @@ def get_configuration_for_impersonation(cls, uri, impersonate_user, username): | |||
return {} | |||
|
|||
@classmethod | |||
def execute(cls, cursor, query, async=False): | |||
def execute(cls, cursor, query, async_=False): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here the async_
keyword argument is not being used anywhere. The reason why it's here is because some engines (actually, only HiveEngineSpec
) have the async
option and override this method. SQL Lab doesn't differentiate between engines, and regardless of the backend will call:
db_engine_spec.execute(cursor, query.executed_sql, async_=True)
It's better to change the method signature here to:
def execute(cls, cursor, query, **kwargs):
This way, any extra keyword arguments are stored in kwargs
, and derived classes can declare more specific signatures, eg, for HiveEngineSpec
:
def execute(cls, cursor, query, async_=False, **kwargs):
@@ -1276,8 +1276,8 @@ def get_configuration_for_impersonation(cls, uri, impersonate_user, username): | |||
return configuration | |||
|
|||
@staticmethod | |||
def execute(cursor, query, async=False): | |||
cursor.execute(query, async=async) | |||
def execute(cursor, query, async_=False): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change this to:
def execute(cursor, query, async_=False, **kwargs):
(See my comment above.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this one be def execute(cursor, query, **kwargs):
? Then downstream systems like PyHive can specify async_
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we had def execute(cursor, query, **kwargs):
here then the method would have to be like this:
def execute(cursor, query, **kwargs):
kwargs['async'] = kwargs.pop('async_', False)
cursor.execute(query, **kwargs)
Since SQL Lab is passing async_=True
, but PyHive expects an async
named argument. We're basically mapping the keyword argument from async_
to async
, as it's passed from SQL Lab to PyHive.
I prefer having it explicitly named in the function signature instead:
def execute(cursor, query, async_=False, **kwargs):
kwargs = {'async': async_}
cursor.execute(query, **kwargs)
This way if you're looking at this line in sql_lab.py
:
You can grep
and find which function takes the async
(async_
) argument.
superset/db_engine_specs.py
Outdated
def execute(cursor, query, async=False): | ||
cursor.execute(query, async=async) | ||
def execute(cursor, query, async_=False): | ||
cursor.execute(query, async_=async_) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here, cursor
is a pyhive cursor, and unfortunately it expects a keyword argument called async
: https://github.com/dropbox/PyHive/blob/65076bbc8697a423b438dc03e928a6dff86fd2cb/pyhive/hive.py#L337
There's an open issue to address this: dropbox/PyHive#148 Unfortunately, until they fix it, we won't be able to run Hive queries in Superset under Python 3.7. :(
In the meantime, I would recommend changing this to:
kwargs = {'async': async_}
cursor.execute(query, **kwargs)
This will allow Superset to work with Python 3.7 with other engines, and with Hive when using Python < 3.7.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For reference: dropbox/PyHive#232
…sync_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6
superset/db_engine_specs.py
Outdated
if cls.arraysize: | ||
cursor.arraysize = cls.arraysize | ||
cursor.execute(query) | ||
cursor.execute(query, **kwargs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should actually be just:
cursor.execute(query)
Otherwise when SQL Lab calls:
db_engine_spec.execute(cursor, query.executed_sql, async_=True)
The async_
kwarg will be passed down to all the cursors, and this could potentially cause an exception if their method signatures don't have **kwargs
.
superset/db_engine_specs.py
Outdated
def execute(cursor, query, async=False): | ||
cursor.execute(query, async=async) | ||
def execute(cursor, query, **kwargs): | ||
cursor.execute(query, **kwargs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This won't work as is, since your PR changes SQL Lab to pass the async_
keyword to the execute
method, and PyHive is (still) expecting the async
keyword. You need to map the async_
that you receive from SQL Lab into async
for PyHive, see two approaches in this comment: #5759
(Sorry if this is confusing, this is a bit of Python black magic...)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments (cherry picked from commit ae3fb04)
* Bug: fixing async syntax for python 3.7 Rename async to async_ so superset installs for python 3.7. * Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6 * addressing additional pr comments
Rename async to async_ so superset installs for python 3.7. 👀 @betodealmeida