google-cloud-ruby does not work with Application Default Credentials

[beccasaurus]

See also: google-cloud-node does not work with Application Default Credentials

Problem

After running gcloud beta auth application-default login I should be able to use google-cloud-ruby to access Google Cloud Platform services, but I can't. I must set the GOOGLE_CLOUD_PROJECT env var or explicitly pass a project ID to a client for google-cloud-ruby to work.

Steps to Reproduce

1. Install the Google Cloud SDK and Ruby
2. Run gcloud init, you'll be prompted to set a default project
3. Run gcloud beta auth application-default login and follow the OAuth 2.0 flow to login
4. Run gem install google-cloud-storage to install a google-cloud-ruby client library
5. Run irb and use the google-cloud-storage gem to interact with Google Cloud Storage. In this case, I am following the example code snippet for Retrieving Buckets found on the website for google-cloud-ruby.

>> require "google/cloud"
=> true
>> gcloud = Google::Cloud.new
=> #<Object:0x007fc27b105f78 @project=nil, @keyfile=nil, @retries=nil, @timeout=nil>
>> storage = gcloud.storage
ArgumentError: project is missing
    from /Users/remily/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-cloud-storage-0.20.0/lib/google-cloud-storage.rb:110:in `storage'
    from /Users/remily/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-cloud-storage-0.20.0/lib/google-cloud-storage.rb:66:in `storage'
    from (irb):3
    from /Users/remily/.rbenv/versions/2.3.1/bin/irb:11:in `<main>'

Expected

After a user runs gcloud beta auth application-default login, google-cloud-ruby client libraries should work without explicitly setting a project ID via the code or an environment variable.

[blowmage]

Thanks for the issue, @remi. I was not aware this was being discussed on the other language projects.

In googleapis/google-cloud-node#1653 it seems like the consensus is that this functionality should be provided by the authentication library, and not the google-cloud-node library. For ruby, that is the googleauth gem. The googleauth gem does sniff for both the GCE environment, and the gcloud SDK Application Default Credentials. But when we asked them to support retrieving the default project_id for GCE they declined, saying that it was not something the library should provide.

What is your opinion? Is this something that googleauth should provide, as it looks to be in future releases for python and node? Or something that google-cloud should provide?

[beccasaurus]

Because this feature is so tightly coupled to the Google Cloud SDK command-line tool, I think it makes sense for this feature to be implemented in google-cloud rather than the generalized Ruby Google API client.

For Python, project ID detection is currently implemented in google-cloud-python, not pushed down into the general Python Google API client.

I rather like Python's project detection:

1. GOOGLE_CLOUD_PROJECT environment variable
2. GOOGLE_APPLICATION_CREDENTIALS JSON file
3. Get default service project from $ gcloud beta auth application-default login
4. Google App Engine application ID (not relevant for Ruby)
5. Google Compute Engine project ID (from metadata server)

Also, I see that googleauth sniffs the Google Compute Engine metadata server for an auth token, which is very relevant to auth, I think that it makes the most sense for google-cloud-ruby to lookup the project ID.

What do you think?

[beccasaurus]

Follow-up comment: I have been discussing this offline with Python and Node.js stakeholders. The referenced Node.js issue also includes discussion about upcoming changes to the the Python Google Auth library to move project detection into the authentication library.

Reasoning / benefits:

1. That Auth Library already has code that is specific to Google Cloud use-cases, such as Google Compute Engine use-cases and default credential lookup
2. Having the code in one place is ideal for if/when any changes are made, eg. to config file format/location/etc
3. Having this logic live outside of an API client library will allow any client-library to make use of this functionality, eg. the existing Google API client for Ruby will be able to make use of this feature as well

The benefits of putting this code into the Google Auth library for Ruby are great and will benefit all libraries that make use of googleauth

I'll follow-up with an issue files on the googleauth library. We can block this pending support for project detection in googleauth?

/cc @jonparrott @jmdobry

[blowmage]

Sounds good to me! We noticed that the newer keyfile JSON files also include the project_id. Since the google-cloud libs already provide googleauth with the JSON file I think it would be great if it gave back the project_id as well as the credentials object.

[blowmage]

This issue is blocked by googleapis/google-auth-library-ruby#90, so I am moving this issue to the feature backlog until we are unblocked.

[AteqEjaz]

Still having the same issue with Ruby version 2.4.4 thanks

[blowmage]

The recently released googleauth 0.7.0 will support the functionality to extract the project_id from service account credentials, but both google-api-client and google-gax pin the dependency on googleauth to a version lower than 0.7.0. We will have to coordinate the loosening of the downstream dependencies before we can start making use of this feature in the google-cloud-* gems.

[blowmage]

I'm going to reclose this issue and open a new issue for the remaining work.