What do you mean "video file that I'm perfectly willing to play in my browser". Isn't it safe to assume that no video file can escape the browser decoding sandbox?
I'm not up-to-speed with the current state of sandboxing in browsers, but in principle it's (on modern operating systems) not especially hard for them to sandbox the decoding into a separate process with basically no privileges beyond rendering a video stream. It's a bit trickier if we're only considering demuxing and delegating decoding to the hardware, but that's a much smaller attack surface.
A manually run ffmpeg on the command line does nothing to restrict its privileges, and its security model has very little interest in doing so, while browsers very much have.
Why would that be safe to assume? If that were a reasonable assumption, you could just as well assume that it's safe to run ffmpeg.