[munificent]

There aren't many significant downsides I know of, it's just really easy to accidentally make your game not deterministic.

You have to entirely isolate the game state from any sources of non-determinism. The latter can include: subtle CPU timing issues, GPU timing, other GPU artifacts, the system clock, timing from IO operations. If you want the state to be deterministic across machines (useful for debugging multiplayer stuff) then you also need to include floating point operations (some chips behave differently on some boundary cases, I think) as well as some graphics operations (thinks like texture operations and rounding are always bitwise identical across GPUs).

If any bit of non-determinism sneaks in from one of these, it will wander through and pollute any other operations and data that depend on it. Flushing out non-determinism bugs can almost feel as difficult as debugging a non-deterministic engine.

I was at EA when the Madden team refactored the engine to be deterministic. It took a full cycle of bug hunting, but it was marvelous once it got there. QA could just send over a replay file and any dev could simply load up the replay and repro the bugs. In fact, the user-facing replay system in the game ("Let's watch that play again in slow mo!") was simply restarting the engine and then replaying the user inputs deterministically to resimulate the whole game again.

[skykooler]

There's a bunch of excellent blog posts from the Factorio developers about tracking down tiny bits of non-determinism (since it uses deterministic multiplayer, as there is way too much dynamic world state to update constantly over the network).