GBQ: Handle deprecation of oauth2client.tools.run()

[jtratner]

Right now, when you use GbqConnector, you get the following warning:

WARNING:root:This function, oauth2client.tools.run(), and the use of the gflags library are deprecated and will be removed in a future version of the library.

I'm not sure how quickly this deprecation will happen or if there are alternatives to keep things the way that they are; but at least worthwhile to have an issue here to address it (in the future).

As an aside, I've been glancing at the source for the bq client that Google provides as part of Google Cloud SDK and I'm wondering how public the methods are (i.e., if we could adapt some of the code or functions that bq.py uses to authenticate and be able to get credentials set up from gcloud auth login rather than requiring a separate oauth flow). Not sure though.

Again, cc @jacobschaer , I'd be interested in your opinion on whether this is worth investigating at all. I've been finding read_gbq and to_gbq quite useful - thanks for putting it together!

[jacobschaer]

Using bq sounds like what we were doing back in v1. In discussions with Google, it seemed that their idea of bq is simply a reference, and the implementation is not to be relied upon in future versions. The decision was made to move away since we had continuous issues with bugs we were finding. At the time, none of the methods were "public" (to the degree with Python allows), and everything was wrapped by Google's apputils making it difficult to access as a library. I'd have to look to see what they intend to use to replace oauth2client.tools.run() - last we were told, Google was committed to a minimum of 1 year support for their API's.

[jtratner]

gotcha - maybe it's just that we need to invoke it differently or something.

On Fri, Sep 19, 2014 at 8:47 PM, Jacob Schaer [email protected]
wrote:

Using bq sounds like what we were doing back in v1. In discussions with
Google, it seemed that their idea of bq is simply a reference, and the
implementation is not to be relied upon in future versions. The decision
was made to move away since we had continuous issues with bugs we were
finding. At the time, none of the methods were "public" (to the degree with
Python allows), and everything was wrapped by Google's apputils making it
difficult to access as a library. I'd have to look to see what they intend
to use to replace oauth2client.tools.run() - last we were told, Google
was committed to a minimum of 1 year support for their API's.

—
Reply to this email directly or view it on GitHub
#8327 (comment).

[azbones]

@jtratner I would be careful with any of the Google client libraries including gcloud. I've been testing a Java library they have which uses gcloud and it apparently uses some custom interval version of boto which conflicted with my desktop python distro's boto library. So, the only way I could get it to work was to run their java code from within a clean python virtual environment. As Jacob said, I would suggest sticking to the APIs.

I'm not sure about the need, but another potential feature for your hackathon might also be some method for authenticating in a server environment. Also, if you want to talk about testing or how we set up the pypandas.t​est@gmail.​com for Oauth, let us know.

[jtratner]

@azbones @jacobschaer I agree - we definitely don't want to rely on some internal and undocumented method to get to authentication.

Right now, it looks like we might actually be able to just pull local credentials like gcloud does, but we could probably just let people specify a path to their credentials file and avoid the need to authorize their credentials to an external application, rather than using Google's helpers for getting to the credentials file. If that's not available, then fall back oauth2client.tools.run

[azbones]

If there is anything we can do to help, just let us know. I think analytics focused services like BQ and pandas are great for developing a powerful analysis workflow.

[jreback]

@azbones the immediate issue that prompted this was #8107

seems that setuptools is a run-time dep to figure out the versions of the various gbq deps.

We don't require this on windows (where it was discovered), and its not a listed dep.

Just trying to avoid adding something else.

[azbones]

Ah, got it and that makes sense. I'll try catching up with @jacobschaer also given he was looking into it.

[aginiewicz]

Any news on this one? I'm packager of pandas on Arch Linux and gflags is only dependency left that isn't compatible with python 3 - and seems won't be in any time soon. Even google-api-python-client
now has python 3 version.

[jreback]

@aginiewicz IIRC gbq is not tested / enabled for py3 at all. I am not sure that status in general.

cc @jacobschaer
cc @azbones

[azbones]

Hey @jtratner, @jacobschaer and I are going to try to look at the following in the next week:

-options for Google's deprecating oauth2client.tools.run() method
-that feature that I think someone was adding to allow server auth
-python 3 support (we had some concern when we first considered this, but we'll look into again)

[azbones]

@jacobschaer @sean-schaefer @jtratner Also, I just saw #9713, so we can look at testing again as we should catch things like that.

[cgrin]

@azbones Just came over to take a look at this and did a bit of digging. It's actually not a huge change, and I've already written a patch to fix it. The replacement for tools.run is tools.run_flow(). Change the import from

from oauth2client.tools import run

to

from oauth2client.tools import run_flow, argparser

and invoke with

credentials = run_flow(flow, storage, argparser.parse_args([]))

Not sure if I should push this to #9722 or wait for that to accept, and make a new pull for this issue.

@jreback @jacobschaer - Thoughts on how to handle this?

[azbones]

@cgrin Cool- that is the same conclusion we had come to! We'll leave it to you, but let us know if you need any help.

The other feature we were thinking about in gbq was #8489 which is generally focused on an improved auth flow for servers. We haven't researched that one yet, but I think someone had taken a cut at it.

cc: @jacobschaer @sean-schaefer

[jreback]

closed by #9722