Leaking authorizations

[jsha]

Hi! We've gotten a couple of reports from hosting providers that integrate with Let's Encrypt that they are hitting the pending authz rate limit, and are using xenolf/acme. I suspect that this is because some code path in xenolf/acme can leak authorizations (i.e., neither fulfill nor deactivate them) when there are some forms of errors. I'm not sure in more detail what those conditions might look like, but would be able to attempt a repro? For instance, you could set up lego to issue a lot of certificates against staging, including some non-existent domains, and log the resulting authz URLs. You could then fetch each of the authz URLs and check if any of them are in "pending" state. If they are, that would indicate an authz leak.

Thanks for implementing this client, it's a very popular way to interact with Let's Encrypt!

[mholt]

What version of lego are they using?

Edit: They just posted it on the forum. 5dfe609

FWIW, I know of a server managing thousands of certificates that is not known (yet?) to have any problems...

[RBeaudoin]

We at Heroku are one of those providers that @jsha mentioned who are having issues with the pending authz limits. I'm hoping that #367 will address these issues as we suspect they are related to bad nonce leaving pending authz requests open, but we forked Lego to allow us to cancel pending authz and are pulling in the upstream change today.

[xenolf]

I was able to repro leaks when the solver itself encountered an early error state. @RBeaudoin which challenge are you using?

[RBeaudoin]

@xenolf we are using acme.TLSSNI01

[xenolf]

@RBeaudoin could you try the authz-cleanup branch against staging to see if it solves it for you?

[RBeaudoin]

@xenolf I can test today and let you know how things work. If it does, it would be a huge help thank you.

[jsha]

@xenolf thanks for working on this! out of curiosity, what are the types of early error state that could cause the solver to leak?

[RBeaudoin]

Update: we haven't seen any pending authz issues yet after ~11 hours of running the branch in production. We also haven't seen any bad nonce issues which is very nice.

Unfortunately, we are still seeing the 429 Too Many Requests despite ee0018c and 45beff7 but I'm happy to open a different issue related to that

[xenolf]

@RBeaudoin thanks for the update! I will merge the changes into master asap. As far as the 429 Too Many Requests issue goes... I don't really know what to do about it. We already do limit the requests per second to under the value boulder advertises. Maybe @jsha could give us more insight?

[trecloux]

I tried to fix the 429 issue, with a dedicated RoundTripper because the current solution is scheduling the goroutines but not the http calls.... but it was a failure :-(

[jsha]

@RBeaudoin could you open a separate issue for the 429s you're seeing and I can provide feedback there?

[RBeaudoin]

Opened #390

[xenolf]

Closed by eb711d3