Let's talk about ROLLBACKS!

Alrighty, I finally got annoyed enough to make another blog post, so here we go.
Let's cover rollbacks vs. lag-based networking for fighting games!

WHAT THE HECK IS "NETCODE"?

"Netcode" isn't really a thing itself - it's a word many people use to describe "the many different ways that video games can handle online play".  This means the actual online play part, during a battle/round/quarter/half/fight/skirmish/puzzle (with optional simultaneous argument over voice chat), as opposed to matchmaking/leaderboards/lobbies/ranking/messaging/voting/replays or any other non-gameplay online thing you can do in a video game.

WHAT DOES NETCODE DO?

The basis of online play is transmitting information about MY game to YOUR computer (or game console, which is a computer), and receiving information about YOUR game back to MY computer.

Information takes time to get places.

If two computers that are miles away from each other need to communicate, that information takes time to be transmitted between them.  (A full discussion of the different types of internet transmission delay is outside the scope of this article, but playing over wifi is worse! :^)  

Though things can be done to minimize the transmission lag, we physically can't eliminate 100% of the delay - there is always some lag present whenever information is transmitted across a distance.  And the bigger the distance, or the worse the infrastructure is, the bigger that lag will be.

The purpose of netcode is to handle, hide, or otherwise compensate for information transmission delay in the "best" way possible for a given type of game.

For more discussion of how games like first-person shooters can hide lag in extra-clever ways, or why servers may be used as intermediate communication points, see the Appendix.  For now, we're going to talk about fighting games, and just about the game itself.

OKAY, SO ONLINE LAG IN FIGHTING GAMES...?

To get it out of the way:  the most compact information games can send between computers is just the player's input at a given time.  Everything else about what should happen can be constructed from the previous state of the game and the new input, which is how games work offline.  :^)  So fighting games only send each other "my current player's input", which is why we talk about "inputs" arriving after a delay.  No other information is necessary, so no extra info is sent.

Let's not even talk about "frames", that mysterious unit of time - let's talk about SECONDS, because it's easier!  Imagine Angel and Blanka want to play a match of Online Punch Game, and Blanka is using his tropical-fruit-powered internet connection which has a delay of two whole seconds.  

(I picked two seconds because it is a huge long time in video game terms - all these problems still exist with smaller delays, and are just as bad, but it's easier to conceptualize "two seconds" as "really bad".)

Now, both games start at the same time - which is a whole different type of magic - and they immediately start telling the other computer about what's happening on their side.

That two-second, fruit-based delay means when the timer is at 90 seconds, Angel's game is getting Blanka's inputs from 92 seconds - two whole seconds ago.  Now we have a problem!  How do we handle that delayed, missing information and still keep the game moving?  Well...

WHY DO DIFFERENT TYPES OF NETCODE EXIST?

The short answer here is that people got cleverer over time.  The earliest online play had the simplest solution, and more intricate solutions were invented later on.  The first type of netcode created was...

LAG-BASED

With a 2-second delay between Angel and Blanka, the simplest solution is for Angel's game to also delay Angel's own inputs by two seconds.  That way, when Blanka's inputs from two seconds ago arrive, the game is also processing Angel's inputs from that same time.  Simple, yeah?

Pros:

• Easy to make.  Super simple, doesn't require altering the game itself in any way - just keep a list of previous inputs to hold them until they are needed.
• Doesn't affect the game's onscreen presentation at all.  The game proceeds at regular speed and nothing looks or sounds any different from offline play.  (Ignoring lag spikes for now.)

Cons:

• Delays your inputs to your own character based on the internet delay to the opponent.  Only one downside, but it's a BIG one.

tl;dr:  Takes all the lag out of the visuals and puts it into the controls instead.

Like that Google Stadia GIF:

here is my Google Stadia review in one GIF.

this is on The Washington Post's Gigabit Ethernet last week. pic.twitter.com/qexEv6vyUD

— Gene Park (@GenePark) November 18, 2019

Let's unpack that "con" a bit.

Fighting game players care about response time - 8 frames of input lag is "unplayable".  (Don't worry if you don't know what that means.)  Under lag-based netcode, if Angel sees Blanka starting to do an attack, and Angel immediately tries to defend, Angel's own "defense" input is delayed by two seconds, which means her character gets hit even though she reacted as quickly as she could.  To properly defend, Angel would have had to guess (defending two seconds early) instead of react.

This is a big deal.

In lag-based games, viable online tactics are often very different from viable offline tactics.  Because your own character is delayed, there are situations that you simply CANNOT react to online, you must guess, where offline you could easily react properly based on seeing what's about to happen.

This very problem - lag-based online play is not a viable substitute for offline practice, even against the same person - led a very intelligent guy by the name of Tony Cannon to come up with the concept of "rollbacks", and create netcode that uses them, which he called "GGPO".  It's important to note this so I'm gonna put it in big text:

ROLLBACKS WERE INVENTED TO SPECIFICALLY IMPROVE THE PLAYABILITY OF ONLINE FIGHTING GAMES.  NOT TO PROVIDE SMOOTH VISUALS.  And they do it!  But only if you care very much about the feel of the game, instead of the visuals, which many casual players don't!  So...

WHAT THE HECK ARE ROLLBACKS?

Let's return to Angel, Blanka, and their two-second delay - but now Online Punch Game uses rollbacks!

Angel's game is still receiving Blanka's inputs two seconds late.  But now, the game saves a snapshot of the match from two seconds ago, as well as all of Angel's inputs since then...and when Blanka's delayed input finally arrives, it "rolls back" to that snapshot of the past, alters history by using Blanka's newly-received old input along with Angel's saved input, then runs itself in fast-forward until it gets back to the present.

As Doc Brown would say, the timeline has now skewed into an alternate reality where we got Blanka's delayed input "on time"!  And the game continues from there.

So what's the big deal now?

The big deal is that now Angel's inputs to her own character can happen immediately, even though Blanka's inputs to her game are still delayed!  Reactions to visual cues are once again possible even in online play.  Because we can go back and alter history, Angel can tell her character what to do in the present, immediately.

In fact, Angel can choose how much delay her inputs have, which in turn changes how much altering of history the game has to do.  If Angel chooses to experience the same amount of delay as the lag-based netcode would need, two seconds in our example, then the game doesn't have to alter history at all and the experience is exactly equivalent to the lag-based netcode.

Pros:

• You can play the game with zero "felt" input delay, bringing online and offline reaction speeds much closer together and making online practice much more useful.
• You can choose the delay you feel on your own character - immediate reaction for better playability, or delayed reaction for better visuals.
• Improved handling of lag spikes - a short period of longer delay doesn't necessarily mean the game needs to stop, or increase your own lag, to continue.

Cons:

• Altering history changes the present!  This is what a "rollback" is, a large visual change between one moment and the next.  The opponent might suddenly be in the middle of an attack, or may have blocked an attack that previously looked like it successfully hit.

tl;dr:  Takes the lag and puts it wherever you want it, either in the visuals or in the controls.

Let's unpack the "con" once again.

We still can't get rid of the lag!  We can either put it in the controls or in the visuals, or a bit of both.

Putting the lag in the visuals can indeed look (or sound) bad!  That's what people who complain about rollbacks complain about:  "I did a move and it hit but they blocked it"/"I did a super but then didn't because my meter wasn't full"/etc.

This complaint tends to be unduly effective at convincing casual players that rollbacks are bad, because bad visuals sure do look bad.  But what it ignores is that you can't SEE input lag in a video, but you sure can FEEL it when you play...and constant input lag is a way bigger barrier to improvement than occasional visual discontinuities.  (Plus, like I keep saying, if you prefer the input lag you can choose it with rollback-based netcode - but your opponent doesn't have to!)

[Technical stuff:  The game continuously saves snapshots, and any time it gets an old input from Blanka it goes back to the appropriate snapshot and alters history, so the "present" is continuously changing.  Since the opponent's inputs are always delayed, this means the "present" you see is a sort of incomplete reality, but in real-time that turns out not to matter for intermediate- or high-level play anywhere near as much as eliminating the delay on your own inputs does.]

SO WHY DO I WANT ROLLBACK NETCODE IN THE GAMES I PLAY?

Simply put, you want rollback-based netcode in your fighting game because you can play either way - with rollbacks, or with lag.  

Different people have different goals, and will want different choices.
If you want to have "game feel" that's close to offline quality but you don't care how the game looks then you choose a lower "my character" delay and more rollbacks, and if you want the game to look nice but don't care how it feels then you choose a higher "my character" delay and no rollbacks.

If the game doesn't support rollbacks, that first option is not available, so everyone suffers the constant lag on their own character, and online play deviates significantly from offline play.

THE USEFULNESS OF ROLLBACKS IS NOT RELATED TO THE DESIGN OR GAMEPLAY OF YOUR FIGHTING GAME!
The elimination of "my character" input lag is equally beneficial in Guilty Gear or Street Fighter or Samurai Shodown or Box Fighter or KI or Melty Blood or Skullgirls or Marvel.  [tm, tm, tm, tm, tm, tm, tm, tm.]

OKAY, I UNDERSTAND WHY ROLLBACKS ARE USEFUL - HOW HARD IS IT TO ADD ROLLBACK SUPPORT IN AN ENGINE?

Rollback support only needs two things from the game engine:

1. The ability to save a snapshot of the game at a particular point in time, and to reload a snapshot later.  (Typically this is accomplished by saving the useful properties of the fighters/projectiles/etc, and not the background.)
2. The ability to run the game in fast-forward, simulating from the altered past up to the present very quickly.  (Typically this is accomplished by just running the game logic, which is very simple for fighting games, without drawing anything or decompressing animations or doing anything visual.)

It does not matter if your game is 2D or 3D, using complicated models with cloth physics or cardboard and duct tape - those are the only two things the engine needs in order to support rollback netcode.  It does not have to be written "from the ground up" to support rollbacks.

Programmers might object anyway, because:

Adding those things, even if they're minimal work, is more work than the "no changes at all" required for lag-based netcode.

-and-

Adding those things can be more difficult depending on the architecture of the engine - however, typically things that make adding rollbacks difficult also make actually-creating-the-game more difficult, too.  For example:

If a 3D game attaches the hitboxes to the character's skeleton, then fast-forwarding the game would involve animating the skeleton over and over to compute the position of the hitboxes...but this approach has so many other downsides (for both development and gameplay!) that it is recommended the hitbox locations be "baked out" and not attached to the final animated skeleton in the first place.  :^P

If the game uses a commercial engine, for example Unreal or Unity, then saving a snapshot might be quite a large undertaking because it will require serializing objects your own way.  Additionally, lag-based netcode may already be built into the engine, which will further encourage engineers to think lag-based netcode is "good enough".

So let me say this ONE MORE TIME:

FOR HIGH-LEVEL PLAY OVER ANY SIGNIFICANT DISTANCE, "GOOD ENOUGH" IS NOT GOOD ENOUGH.

If you want your players to play their best, especially if they live in America or Europe where larger distances and worse infrastructure is common, lag-based netcode is SIMPLY NOT GOOD ENOUGH.

APPENDIX: DIFFERENT TYPES OF NETWORKING SOLUTIONS / WHAT'S THIS ABOUT "SERVER-BASED," "RELAY SERVER," OR "PEER-TO-PEER"?

[This part is for nerds, or aspiring nerds.  I know because I am one and I wrote it.]

For a two-person game, you just need to get information between two computers.

The fastest way to get information from Computer A to Computer B is by sending it directly between them.  This is called a peer-to-peer connection, and is ALWAYS preferable for 2-player fighting games when it is available.

Having any kind of server in the middle IS SLOWER.  Imagine sending a letter to your friend directly, or sending a letter to your friend by sending it to your grandma and having her forward it to your friend.  The grandma route will be slower, even if she notices the letter right away.  (Please don't be a smart alec in the comments by positing what if your grandma lives next door to your friend, or is The Flash(tm), or something.  You understand the example.  Thanks.)

There are situations where servers are useful in two-person games, but they aren't for speed!

A relay server is a server that is used simply to repeat information because Computer A has trouble reaching Computer B.  If Computer A can't talk to Computer B because of a problem with the intertubes, Computer A can talk to a relay server which then talks to Computer B.  This is only preferred because otherwise A and B can't talk at all; it does not improve the connection speed.

I've also heard an amazing theory, paraphrased as, "Amazon(tm) promises 40ms ping times from anywhere in the world to their cloud, so I'm gonna use Amazon as an intermediary and beat peer-to-peer ping times".  How do I even explain the problem here...that's like saying, "I live five minutes from the freeway, and my friend lives five minutes from the freeway, so we live ten minutes from each other even though I live in California and my friend lives in New York."  You still have to drive on the freeway.

If my ping time from LA to NY is 200ms, that's because there's a huge distance to cover.  If my ping time in LA to Amazon is 40ms, and the ping time from NY to Amazon is 40ms, we're not talking to the same server.  Those two servers still have to communicate, and THAT ping time is going to make up the rest of the 200ms...or more.

(Unless Amazon also built a super-secret, orders-of-magnitude-faster-than-the-regular-internet secondary worldwide network for their cloud to use, of course.  Which is not entirely impossible because Amazon has so much money that they should just give me $20m to make games because this article was so useful and mentioned them, hey ya hear that?  ...anyway...)

So for fighting games, speed and low latency matter, and you can get away with not having a server unless you need it for connectivity purposes.

BUT

For a three-or-more-person game, servers are not only useful, they're preferred.

If A, B, and C are playing a game and you want to do peer-to-peer connections, that's A-B and A-C and B-C that all need to send information; if you add D, then you have A-B, A-C, A-D, B-C, B-D, and C-D, and your game's performance will be dictated by the single worst connection out of all of those, and if any of them fail you can't play

.

By contrast, with a server you have A-S, B-S (heh), C-S (heh), and D-S (heh); each player only adds one connection, and the server can be put on a super good network connection for itself.  Additionally, if servers are available in many locations (hi Amazon!) then once you know all the players the game can choose a server that's somewhere between A, B, C, and D, and improve everyone's ping time to the server all at once.

As well, with the server model there is ONE authoritative computer that knows exactly "what the game is really like" - the server.  It can tell if you try to teleport or whatever, and because it is the one who decides what's "real" it becomes much harder to cheat.  Still possible, depending on where authority really is held, but much more difficult.

AND

FPSes can cheat.  A lot.  I don't mean cheat like cheat, I mean cheat like "hide lag by faking stuff and nobody can ever know".

In fighting games, your character's position matters a great deal.  Hits are directly attached to you, there's nothing between you and the opponent but your fists.

In FPSes, though, there's a bullet between the two of you.  And a server.  And combining those two things lets the game stretch the truth to players.  Very convincingly!

If I'm in the kitchen and I see you in the kitchen and I shoot you, and the server tells me I shot you (because I shot at where I saw you) and also tells you I shot you, then you got shot.  But if you were "really" outside on the lawn...nobody knows that, AND it didn't matter for all that information in the first sentence!  I saw you in the kitchen (because of lag), I shot you where I saw you, and you got shot.  The only things that would know the difference are the bullet and the server, and neither of those are the players so both of them can ignore the inconvenient facts.

Fun fact - this is a major reason why melee hits in FPSes are so much more frustrating to use than melee hits in fighting games - a "hit" in an FPS is a big stationary 'bullet' that your weapon makes, and in THAT CASE - because it's stationary and comes from your real position - the opponent's real position does matter, which may not be accurately reported to your game right this second so you may be seeing something different, and that's why they miss (or hit) in cases where you don't think they "should".

It doesn't mean people are cheating or the games are bad, but especially with bad ping times FPSes do this more than people would like to admit.  :^)  And fighting games can't do that, period.  :^P

I think that's everything!