API Limit

[roderik]

I've been playing with boxes for an afternoon, and got into some issues fairly soon. Just found out why: API Rate Limit Exceeded in librarian. If i can hit it this fast by myself and one VM, it will go down quickly when i start rolling this out at work with 20 developers on one external ip address.

Is there a way to circumvent this limit in boxen? Seems like librarian itself has support for api keys https://github.com/rodjek/librarian-puppet/blob/master/lib/librarian/puppet/source/githubtarball.rb#L115 but is there a way to integrate this into boxen?

[wfarr]

We actually do use librarian-puppet's support for this: https://github.com/boxen/boxen/blob/master/lib/boxen/puppeteer.rb#L79-L86

The rate limiting with an oauth token as per above is 5000 req/user/hour: http://developer.github.com/v3/#rate-limiting

Rolling this out across 20 developers shouldn't be a problem as on first-run they'll each get an oauth token generated and that token will be used for pulling down modules if needed. Referring to #48, while you shouldn't hit rate limits because of the per-user tokens, checking in the cached module downloads before rolling it out to your developers will guarantee you don't run into that.

[roderik]

ah, i see what the problem is then, i've been running librarian by hand to debug something, not using the oauth token and hit the 60 reqs limit quickly.

Maybe it's a good idea to add --verbose to https://github.com/boxen/boxen/blob/master/lib/boxen/puppeteer.rb#L84 if config.debug? is true? There is no other way to debug what librarian is doing from within boxen

[wfarr]

I think that's a good idea. I'd be happy to merge a Pull Request to that effect.

[all9lives]

I think I'm still running into the rate limiting issue, even though boxen asks me for my github credentials when it first starts. Is there a way to check what oAuth token is being used? Or to set it manually?

From my experience, if I wait a couple of hours, the rate limit is reset, but if I install more than one new module at a time I run into this issue.

aman:lv-boxen jeremy$ script/boxen --debug
Boxen has a dirty tree, won't auto-update!
Could not resolve the dependencies.
Can't run Puppet, fetching dependencies with librarian failed.
aman:lv-boxen jeremy$ 

[harperreed]

I am also hitting this limit. I worry that if I were to onboard more than one person, this would be a problem.

[timshadel]

While the symptoms were the same for me, the underlying cause of the issue was slightly different. Here's my output with some extra debug thrown in:

API call: https://api.github.com/repos/boxen/puppet-boxen/tags
API call failed (403): {"message":"Missing or invalid User Agent string. See http://developer.github.com/v3/#user-agent-required"}
Unable to find module 'boxen/puppet-boxen' on https://github.com
/opt/boxen/repo/.bundle/ruby/1.8/gems/librarian-puppet-0.9.8/lib/librarian/puppet/source/githubtarball.rb:22:in `versions'
/opt/boxen/repo/.bundle/ruby/1.8/gems/librarian-puppet-0.9.8/lib/librarian/puppet/source/githubtarball.rb:39:in `manifests'

The underlying issue for my install was the Github API change to require user agent strings. The fix:

# /opt/boxen/repo/.bundle/ruby/1.8/gems/librarian-puppet-0.9.8/lib/librarian/puppet/source/githubtarball.rb
request = Net::HTTP::Get.new(uri.request_uri, {'User-Agent' => 'librarian-puppet-0.9.8'})

Boxen is forcing ruby 1.8 at /opt/boxen/repo/.ruby-version, so it doesn't get the default UA string that Ruby has in 1.9.3 and later, so a local file edit is the fastest way to move past the issue.

[wfarr]

Tim — Update librarian-puppet in the Gemfile to get the fix in 0.9.9

On Wed, May 1, 2013 at 9:59 AM, Tim Shadel [email protected] wrote:

While the symptoms were the same for me, the underlying cause of the issue
was slightly different. Here's my output with some extra debug thrown in:

API call: https://api.github.com/repos/boxen/puppet-boxen/tags
API call failed (403): {"message":"Missing or invalid User Agent string. See http://developer.github.com/v3/#user-agent-required"}
Unable to find module 'boxen/puppet-boxen' on https://github.com
/opt/boxen/repo/.bundle/ruby/1.8/gems/librarian-puppet-0.9.8/lib/librarian/puppet/source/githubtarball.rb:22:in versions' /opt/boxen/repo/.bundle/ruby/1.8/gems/librarian-puppet-0.9.8/lib/librarian/puppet/source/githubtarball.rb:39:inmanifests'

The underlying issue for my install was the Github API changehttp://developer.github.com/v3/#user-agent-requiredto require user agent strings. The fix:

/opt/boxen/repo/.bundle/ruby/1.8/gems/librarian-puppet-0.9.8/lib/librarian/puppet/source/githubtarball.rb

request = Net::HTTP::Get.new(uri.request_uri, {'User-Agent' => 'librarian-puppet-0.9.8'})

Boxen is forcing ruby 1.8 at /opt/boxen/repo/.ruby-version, so it doesn't
get the default UA string that Ruby has in 1.9.3 and later, so a local file
edit is the fastest way to move past the issue.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/50#issuecomment-17291867
.

[timshadel]

Nice. I didn't catch the update with #259 that solves this. Merged it in and I think I'm set now. Thx! :-)