github rate limit is hit pretty quickly

[leotaku]

When I try to update all dependencies in multiple large projects, it is realistic for the Github API calls to hit a rate limit.

This is of course a problem, because it essentially "blocks" me from working with niv for 1 hour. Maybe there is something that could be done?

Relevant link: https://developer.github.com/v3/#rate-limiting

[nmattia]

(I was on holiday and missed the notification, apologies)

Yeah that’s indeed an issue. The best (only?) solution would be to add support for a GITHUB_TOKEN environment variable. To clarify, you would hit the same problem when doing it by hand (without niv), right?

[leotaku]

(I was on holiday and missed the notification, apologies)

No worries, thanks for responding.

To clarify, you would hit the same problem when doing it by hand (without niv), right?

Could you specify what you mean by "it"? Sorry if I'm missing something obvious here.

[nmattia]

Could you specify what you mean by "it"? Sorry if I'm missing something obvious here.

I wasn't very clear. I meant "doing it by hand" as in not using niv, e.g. with a bash script that calls GitHub's API to check for the latest rev. Of course this is not a problem if you copy the rev by hand from the GitHub project page, since most likely you'd be logged in.

[leotaku]

In that case, yes I would suspect that any application that uses the API in this manner would hit the rate limit in the same way niv does.

There however seems to be a feature for conditional requests. It might be possible to store the ETag header and then use If-Modified-Since for subsequent API calls.
That way, if one dependency has received updates only the API calls needed for said dependency count towards the rate limit. (I have not tested this, though)

[nmattia]

Actually I think it might be solved by #58; hopefully fetGit and fetchFromGitHub use SSH to login

[leotaku]

@nmattia Would this switch also replace the code niv currently uses to check if a github repo has been updated? The last time I tried to use fetchFromGitHub as a fetcher in scripts using symbolic revs like master, its mechanism for that was extremely unreliable and slow.

(For that reason) I think fetchFromGitHub/fetchGit are really only meant for static revs, what is your plan for implenting nivs update functionality through them?

[leotaku]

I'm assuming you want to reimplement the update functionality using the fetchFromGitHub fetcher, because otherwise niv itself would still have to make calls to the GitHub API itself, which means the problem wouldn't actually be solved.

But maybe I am just misunderstanding your intentions.

[nmattia]

@leotaku re your last comment: you're completely right, it was a mix of a brainfart and wishful thinking on my end

[madjar]

I'm starting thinking about this, and I'm wondering if using the graphql api wouldn't allow to

• Limit the number of calls to 1 per repo
• Batch the calls for all the repo to a single call, provided a way to introduce that without breaking the architecture (I'm thinking about techniques similar to haxl, monad-batch and friends for that)

I'll need to get acquainted with the Update mechanism, though, but that's probably doable.

[nmattia]

Oh that's a very neat idea. The Update is haxl-ish, so there may be a simple way of implementing this. I don't know graphql very well though, could you give an idea of what this would look like in terms of curl?

[madjar]

Indeed, it looks like it wouldn't be too much effort to add a constructor to Update for batchable queries.

I went through the code, but I did not completely get all the intricacies of it at the first go. I think you mentioned you wanted to write some documentation about; I'd be very curious to read it.

---

For graphql, I crafted the following query to illustrate a few things (using https://developer.github.com/v4/explorer/):

query { 
  niv: repository(owner: "nmattia", name: "niv") {
    description
    homepageUrl
    defaultBranchRef {
      name
      target {
        oid
      }
    }
    ref(qualifiedName: "nm-wip"){
      target {
        oid
      }
    }
  }
  nixpkgs: repository(owner: "nixos", name: "nixpkgs") {
    description
    homepageUrl
    defaultBranchRef {
      name
      target {
        oid
      }
    }
  }
}

• We can get both the description, and defaultBranch, and the commit id (oid) of any branch in the same request (though now I'm not sure it's actually useful to do both at once).
• We can query multiple repositories at once.

The result is:

{
  "data": {
    "niv": {
      "description": "Easy dependency management for Nix projects",
      "homepageUrl": "https://github.com/nmattia/niv",
      "defaultBranchRef": {
        "name": "master",
        "target": {
          "oid": "eb859c309e38450d70d458cf5ce06e6a2046f966"
        }
      },
      "ref": {
        "target": {
          "oid": "300e21f722d4d02376a9f40176fd80b36dfaa6cb"
        }
      }
    },
    "nixpkgs": {
      "description": "Nix Packages collection",
      "homepageUrl": null,
      "defaultBranchRef": {
        "name": "master",
        "target": {
          "oid": "7cb82ca2d369018bf488c06f15e8336b8d07c39f"
        }
      }
    }
  }
}

Here is what the curl looks like:

curl https://api.github.com/graphql -H "Authorization: bearer $GITHUB_TOKEN" -X POST -d '{"query": "query {    niv: repository(owner: \\"nmattia\\", name: \\"niv\\") {     description     homepageUrl     defaultBranchRef {       name       target {         oid       }     }     ref(qualifiedName: \\"nm-wip\\"){       target {         oid       }     }   }   nixpkgs: repository(owner: \\"nixos\\", name: \\"nixpkgs\\") {     description     homepageUrl     defaultBranchRef {       name       target {         oid       }     }   } }"}' https://api.github.com/graphql

That being said, I now realize it's a bit cumbersome, and it only for authenticated users, which might very well be a deal-breaker

[nmattia]

Sweet. Any idea how the rate limiting is applied in that case? missed this part:

only for authenticated users

Re. the doc: bumped priority on the todo list

[madjar]

Out of curiosity, I still looked at how the rate limiting works, and it's quite complex :)
The rate limiting is at 5000 points per hour, where the value in points of a query depends on the complexity (as an exemple, the above query still counts as 1 point).

But yeah, without anonymous access, this is moot.

[madjar]

I've felt into a rabbit hole, and another bad option to go around rate limiting is to use the git protocol instead of the api.
curl 'https://github.com/nmattia/niv.git/info/refs?service=git-upload-pack'

[dysinger]

I've hit this also but I reported it under issue #105 because I couldn't see what was going on and the error message was bunk.

[nmattia]

FWIW now using git sources (niv add git -n foo [email protected]...) should help. I'll move the GitHub updater to use git sources by default.

[psionic-k]

Although this is for cachix github action, uses same nix.conf

    - uses: cachix/install-nix-action@v15
      with:
        extra_nix_config: |
          access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}

https://github.com/cachix/install-nix-action#usage-with-flakes

Don't know why I never get this issue locally unless it's using ssh. For CI with Actions, there's a token available. Might as well. Otherwise make a machine user and create a token or use personal / work account if you need to set this for local development.