Networking In Jumpy

[zicklag]

One of the most difficult things to get working with the recent Jumpy rewrite was networking. It was the first time I had done game networking before, and I learned a lot from it. It took a couple of different attempts with different networking models, and we may still tweak the current design, but here I'm going to give an overview of how networking works in Jumpy today.

ℹ️ Note: This is meant to give a high-level overview of networking to people who may not be familiar with some of the techniques. If you want a more technical overview of how networking works in Jumpy, you can also see our network module in our architecture documentation.

Matchmaking

The first thing to cover is matchmaking: the process of connecting players who want to play a network game with each-other.

The jumpy matchmaker is extremely simple. First, though, lets talk about how it communicates with the game.

Network Protocol: QUIC

All network communication between the game and the matchmaker happen over the QUIC protocol. QUIC is a relatively new network protocol that is built on UDP.

QUIC is great for games because with it you can:

• Send unreliable UDP packets: This is important for high-speed game networking.
• Send reliable streams/messages: This is really useful because we get to have reliability if we need it, without having to resort to using TCP or our own reliability layer.
• Handle multiple connections on the same port: This helps a lot with the matchmaking server, because we can communicate with any number of different players all at the same time, without having to open a large range of UDP ports.

Matchmaking Logic

The matchmaking process is easy:

1. Wait for players to connect
2. Users tell the matchmaking server how many players they want to play with, and what game mode they want to play.
3. The matchmaker adds that player to a waiting room for that game mode and player count combination.
4. If there are enough players in the room, the matchmaker starts the match.

Once the match is started, the matchmaker acts as a proxy, so every time the game needs to send a message to another player, it sends it to the matchmaker and tells the matchmaker what player should get the message.

The cool part is that the game doesn't need to open up any new connections to talk to the players in their match. Everything goes through the matchmaker that they are already connected to, which saves us all kinds of possible trouble we could have trying to connect players directly to each-other.

Synchronization

Now we have our players all connected to each-other and joined to a match through the matchmaker. The next step is to synchronize the game between all the players. This is the hard part!

Peer-to-Peer Rollback

In Jumpy we use a technique called Peer-to-Peer Rollback, powered by the GGRS library, which was itself inspired by GGPO.

The core idea behind the rollback networking model is that if the gameplay is deterministic, and you make the same controller inputs, on the same game frame, then the same thing will happen. Even if you have the game running on different user's computers, if we all put in the same controller inputs, the same thing would happen.

This is where "peer-to-peer" comes from: each player sends all of their network inputs to each-other, without the need for a centralized game server.

So then all we have to do is send the user's inputs across the network?

Not quite. Unfortunately, because of the limits of the universe ( and our networking equipment ), we can only send the user inputs so fast. We can probably expect a 300ms delay on average. So if we needed to get everybody's user inputs before we started the next game frame, we would only get 3 frames-per-second, which would be unplayable. 🙁

The trick is to lie to the player. That sounds bad, but essentially every networked game you ever play needs to lie to the player in one way or another to make it look like you are playing a local game, despite big latency from all players involved.

With p2p rollback we lie to the player by letting the play the game without knowing what the other players are going to do yet.

For example, lets say you are playing the game, and you are on frame 1, and the controller inputs for you and your friend Allice have come over the network already, so we can simulate frame 1 and display it. But now we need to play frame 2. Since your controller is plugged right into your computer, we already have your input, but we don't have the controller input from Alice yet, and it's going to take another 300ms to get here, so what do we do?

We continue to play the game without Alice's input. This is what we mean "lie to the player". We are going pretend that the game just goes on without Alice for just a few frames, until Alice's input comes over the network. So we play the game like Alice isn't doing anything for the next 5 frames until finally, Alice's input for frame 2 comes over the network.

Then comes the fancy part: now that we have Alice's input for frame 2, we rollback to frame 2, and run the frame again, now with Alice's input. Now we know what actually happened on frame 2!

Before we display the frame to the user again, though, we also go through and re-simulate frames 3, 4, 5, 6, and 7, so that we are back up to the frame we were on before we rolled back. The user only sees the game after it's rolled back, re-simulated, and finally caught up to the present.

Now that Alice's input has been applied, Alice might visibly skip a little. Previously we played those frames like Alice wasn't moving, but now we've gone back in time and updated what Alice actually did.

As we roll back and fast forward, we may have to run up to 8 simulation frames every time we show a new frame to the user. But, by using this strategy, we can play the game at a reasonable frames-per-second, and make it fair by including everybody's input on every frame!

Challenges

There are a couple challenges to the Rollback networking model.

Determinism

Deterministic gameplay can be difficult in certain scenarios. Things like multi-threading and certain kinds of floating point number operations can be non-deterministic and give different results even with the same inputs.

In Jumpy, which is built on the Bevy game engine, we have to be careful with certain things like looping over entities that are not guaranteed to happen in any particular order, and other possible causes of not-determinism.

Snapshots

We also have to be able to take a snapshot at the game at every frame, and then have the ability to go back and restore that snapshot when we rollback. This is another thing that isn't super easy in Bevy.

We have it working today, but we also are looking to improve the snapshot and restore performance.

We need to be able to run up to 8 snapshots, 8 simulations, and 1 restore every time we display a new frame to the user. Currently the game isn't performant enough to do all that at 60 FPS, so we temporarily turned the frame rate down to 45 FPS.

Future improvements

Currently, networking in Jumpy is work-in-progress and there are definitely some things we have to improve. Turning the frame rate back up to 60 FPS is a big priority. We also might be able to improve frame skipping over high-latency connections by visually smoothing out the movements of remote players, or guessing their movements so that they don't skip as much.

While Jumpy is primarily meant for "couch play" where you can play with your friends in the same room, we're excited to have a proof-of-concept for online play, and we'll keep learning and improving as we go forward.

Stay tuned!