Allows the usage of Hive Metastore #1533
Conversation
| import hive_metastore.ttypes | ||
| partition_str = self.partition_spec(partition) | ||
| thrift_table = client.get_partition_by_name(database, table, partition_str) | ||
| except hive_metastore.ttypes.NoSuchObjectException as e: |
There was a problem hiding this comment.
can you fix this flake8 issue?
./luigi/contrib/hive.py:187:71: F841 local variable 'e' is assigned to but never used
(btw you can run tox -e flake8 locally to test)
There was a problem hiding this comment.
Sure, I can fix that.
@erikbern This PR requires either the Hive python libs in the pythonpath or hive-thrift-py package installed. Do you have any idea on how to handle this?
There was a problem hiding this comment.
I think you should just leave it as it is. Any dependency in luigi.contrib shouldn't be in requirements.txt. Optionally throw a warning if the right package isn't installed (and you should probably add a note that installing hive-thrift-py is highly recommended)
There was a problem hiding this comment.
Ok, I can do that.
But that's the reason Travis tests are failing as well.
There was a problem hiding this comment.
Where do you see that? When I look at the Travis tests, all I see is the error above
./luigi/contrib/hive.py:187:71: F841 local variable 'e' is assigned to but never used
There was a problem hiding this comment.
You can also add test time dependencies in tox.ini:
There was a problem hiding this comment.
My mistake. You're right. Previously there was an error, but my last commit had fixed it. I thought it still had something to do with the libraries, but if I had checked travis sooner, and seen it was this simple, I would have fixed it earlier.
Allows the usage of Hive Metastore
|
thanks! |
The usage of Hive Metastore greatly speeds up table and partition lookup in luigi.
Although the MetastoreClient was already implemented, the code always looked for one of two alternatives of the hive command, which is slow.
This PR's settings are only active if
is set