I think the difference is that in a video game you are in one location only at any given moment and things travel only forward in time. We can view from any location at any time in volumetric video.
In a lot of racing simulators you can change the position of the "virtual camera". It can be in the cockpit, on the hood, behind the car and on some games in an arbitrary position. Usually replays allows you to see from other competitors and where TV cameras would seat in real world.
CS:GO, TF2, GTA5 and Trackmania (and likely many more) have replay systems where you can pause, play and rewind with a freefly camera. Lots of games have a rewind mechanic: Grid, Baba is You & Viewfinder come to mind. Others have a "Photo Mode" where you can pause with a freefly camera: Starfield & Witcher 3 come to mind.
Valid, yeah. It occurs to me though that the difference is we are making a representation of the real world that can be manipulated like such, as opposed to a simulation of a fabricated world.