"How I Didn't Clip the Sound Effects" Redux - Handling Sounds During Rollbacks

Handling sound effects during rollbacks is a difficult problem!

Now that all the issues with sounds-during-rollbacks are resolved in Skullgirls, I'm gonna document what I did so that everyone else doesn't have to figure it out from scratch.

The title is a reference to Mauve's now-deleted article about Rollcaster, titled "How I didn't clip the sound effects" (archive.org) which is incomprehensible.  I feel okay saying this because when I started working on sounds-in-rollbacks I talked to Mauve, who said, and I quote, "This is incomprehensible, why did I write it like this?"  :^)

Now, I lied - handling sound effects during rollbacks is fairly simple.  You just have to have someone else think about it for a long time first to come up with the simple way.  :^P

Many systems in games, like well-written menus or even the characters in FPSes(!), sometimes work by keeping two states:  a "desired" state, and an "actual" (or "displayed") state.  The "desired" state is "what does the game want the object to do" and the "actual" state is "what is really being shown to the player".

For the menu example, let's say an options menu, there's a selection cursor and you can do things on a menu item like accept/cancel/left/right.  The game operates behind the scenes, moving the cursor and changing option values according to where the game thinks the cursor is, changing the 'desired' state of the menu.  The menu looks at where the cursor wants to be and updates itself by playing animations to move it to where it should be, changing the 'actual' state of the displayed menu.  In a system like that, it doesn't matter if there are long animations for the menu opening/closing/moving the cursor/whatever, since the player doesn't have to wait for those animations to complete before being able to affect the real state of the menu.  I'm sure everyone has experienced menus with really frustrating wait times before you can do anything or after you've changed something...well, that's the way to avoid them.  :^P

Okay, so on to sounds.  Simplistically, games have a number of sound 'channels', each of which can be playing a single sound effect at a time.  Saving/restoring the state of these channels involves knowing about the actual audio hardware, and that's really hard, so we don't want to have to care about that.  :^)  We're gonna assume that the only things we can do are play a sound from the beginning or stop an already-playing sound.  Those are pretty simple things to do, and every sound architecture supports them.

So, wat do?

To start, keep a 'desired' and an 'actual' state for each sound channel.  You want these to include as much information as possible, like which sound is playing (or that none is!), which object played it, and most importantly which frame it was played on, and optionally what number sound it was in that frame.

Next, when you play a sound in the game, instead of really playing the sound on the channel right then, you just update the 'desired' state for that channel with the info you want it to play.  Then, at the end of each frame, sync the state of the sounds:  if the 'actual' state of each sound channel isn't EXACTLY the same as the 'desired' state, stop what's playing, play the new 'desired' sound, and update the 'actual' state to match. That includes stopping what's playing if the actual state is "playing something" and the desired state is "playing nothing".  (Keep in mind that if you stop a sound channel, that should be stored as "played nothing on frame ___", rather than just clearing the state of the channel!  That's important, because it lets us know the frame number on which the "nothing" was played.)

Doing that shouldn't change anything about how your game sounds, it just adds a layer of abstraction so we can have a place to make some decisions.

Once that's in place, it's time to handle rollbacks!

1. Make sure that the 'desired'->'actual' state sync only happens after non-rollback frames, not during rollbacks.
2. Save the 'desired' state of each channel in your savestates, and load them accordingly when a state is loaded.  (If nothing changed, this won't play a new sound because the 'actual' and the reloaded 'desired' will still be the same.)
3. Keep the earliest frame number of all the rollbacks that happened this frame.  (If you're on frame 10, and rolled back to frame 6 and resimulated, that'd be "rollback earliest frame = 6" and then "current frame = 10".  If you didn't roll back since the last time sounds were synced, then earliest frame = current frame = 10.)
4. Add some extra conditions when you sync 'actual' to 'desired', if the states don't match:
• If the 'desired' state's played frame is OUTSIDE the [earliest, current] range, DO NOT play the sound.  The sound couldn't have been played by any rollbacks, so nothing changed from what 'really happened' before - leave the sound channel alone, whether it is playing or not.  This is important step 1.
• However, if this ^ is true and the 'actual' state is INSIDE the [earliest, current] range, then STOP the currently-playing sound!  This means the channel is currently playing a sound which was erased by the rollback, since there is no desired sound on it from inside the rollback range.  This can happen if e.g. someone dodges a hit at the last second:  the character gets hit, and then it rolls back to them not being hit, so you want to cut off the hit sound.  This is important step 2.
• And finally, set the 'desired' state to the 'actual' state, the opposite of what usually happens!  Because what's currently playing is now what's desired, even silence, we need to keep that info around for any future savestates.
• Otherwise, if the 'desired' played frame is INSIDE the [earliest, current] range, sync 'actual' = 'desired' as normal and really play the sound.

And that's it!  It handles every case.  No complicated saving of audio-hardware state, completely platform-agnostic, simple to follow and simple-ish to debug.  

Plus it has the bonus of not skipping short sounds which would otherwise be completely "swallowed" by longer rollbacks if you saved/restored the actual hardware state...which is technically incorrect behavior but turns out to be much more useful for players.

I mostly wrote this for my future self.  :^P  I hope someone else finds it useful!