guest account registration sometimes wedges after a restart

[richvdh]

at restart, we try to calculate the next generated user id, which requires a table-scan which may take many seconds. If we don't manage to correctly calculate it before the next request turns up, then we end up stuck in a loop where both requests repeatedly get allocated the same user id and we never manage to break out of it

[richvdh]

for future reference: the symptoms of this happening include lots of matches for:

grep 'Received response to .*/register_user/%40[1-9].*: 400' var/client_reader2.log

[richvdh]

this is trivially fixed with a lock...

[richvdh]

fixed by #6161

[richvdh]

apparently not fixed

[richvdh]

There are a couple of problems with the current implementation:

• find_next_generated_user_id_localpart always returns zero. Despite going to a great deal of effort to pull all of the numeric user ids out of the database (which is an operation that takes a great deal of CPU and memory), it does not pull out @0:matrix.org, so assumes it is free. We therefore try to register every single userid in turn, starting at zero, until we find a spare one. This takes ages, and leads to a significant amount of hammering on the master.
• But wait! there's more! register_with_store has a ratelimiter which causes it to raise an exception if you call it too often, which makes this loop tight-loop, blocking the entire reactor until the ratelimit clears.

[richvdh]

hopefully fixed again by #6338