[dylan604]

If you've played around with video formats long enough, you'll have seen something like this. This is the basis for most speed change "filters". Only the high end ones do any kind of pixel based motion estimation so that super slo-mo does not look like a slide show.

Also, it's not uncommon to get odd frame rates in the containers. Even on things as "innocent" as listing the frame rate as 29.97 vs 30000/1001 will affect timing (depending on usage). The variations on 23.976 is fun too: 24000/10001. 2997/125.

The muxer is an important step. When using software decoders, things can be a lot more flexible. Back when shiny round discs were popular, there were verifiers that ensured your muxed data was correct. When your decoders are in hardware, there is a very strict set of parameters the input is expected. Any deviation means the hardware cannot play the video. Early days of "cheaper" DVD software had issues with the muxing.

[kmeisthax]

Video editors (or at least Adobe Premiere) have similar problems: they ignore the timestamps entirely, and any clips you import into them will desynchronize unless you've either recorded them from a known-good source with a constant timebase, or re-encoded them at a constant frame rate.