Add Gameserver Allocated health check

[XAMPPRocky]

Is your feature request related to a problem? Please describe.
Talking with @markmandel this issue is a follow on from #279 #404 and #2396. The problem being that currently if your gameserver requires significant time from its GameserverAllocation to when it's ready to accept players, by default you can run into a race condition, where the matchmaker has given the client the server information, and the client attempts to connect before the gameserver is actually ready to serve the connection, causing the client to be rejected.

This can be worked around by adding a label or annotation that indicates when a server is really ready or implementing it in the connection logic itself, however the downside is that is a game specific solution, and requires any and all services which also depend on knowing when a gameserver is ready to be played such as Quilkin to also know how that specific game's logic works to be considered playable.

It would reduce a lot of effort and allow other services to leverage the Allocated state if it could be a strong guaranteed that when a server is Allocated it's always playable for players.

Describe the solution you'd like
The solution that I've come with is to add a new state to the gameserver CRD, called Allocating, and add a health check for determining when a gameserver has become Allocated.

Instead of agones-controller immediately setting the gameserver to Allocated, it would instead set it to Allocating, if no allocating health check is set, then the existing behaviour is triggered, and the server is immediately set Allocated.

If there is an allocating health check, then the agones-controller would begin to poll the health check to determine when the server has allocated (with a final timeout to prevent a server from being stuck forever Allocating).

Describe alternatives you've considered
How the gameserver communicates with agones could be a polling health check or it could be a webhook or Server Sent Event, the source of the health check isn't so important, whichever is the most convenient and compatible for most of the server SDKs would be fine.

[github-actions]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[XAMPPRocky]

Still an issue.

[github-actions]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[XAMPPRocky]

still an issue

[github-actions]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[XAMPPRocky]

still an issue

[github-actions]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[XAMPPRocky]

Still an issue

[github-actions]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[XAMPPRocky]

Still an issue

[github-actions]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[github-actions]

This issue is marked as obsolete due to inactivity for last 60 days. To avoid issue getting closed in next 30 days, please add a comment or add 'awaiting-maintainer' label. Thank you for your contributions

[XAMPPRocky]

Still an issue

[lacroixthomas]

Hello,
I'm interested to work on this feature, but I might gonna need some more info regarding the health check part, will come back to you with some questions / suggestions about it

[markmandel]

I think first step would be to design how this would potentially work. To be fair - this has come up multiple times over the years, and I've (or anyone else) has never been able to come up with a design that fits within the constraints of the system, and wasn't just a reimplementation of a K8s watching system -- and also, having long term blocking APIs is generally not a great idea.

But definitely open to new ideas! @XAMPPRocky is pretty passionate about it, otherwise I'd likely push to close this as a "won't fix", but I'd push for taking this as a last chance to try and come up with a design where this may work, or probably put it to bed for good.

Sound good?