Video compression is typically not about "take each frame, and make it as small as possible". Video compression revolves around cleverly making use of deltas and segmenting the image so that you can reuse as much data as possible from previous frames/keyframes.
2D image compression is missing a dimension. Much of the savings to be had in a video compression algorithm are going to be based upon the similarity between frames (ie. across time) rather than within a given frame. Therefore, an algorithm that is designed only for the 2D image is not going to deliver the goods when applied to video. An example would be animated GIF. It's terrible.
I didn't read the spec, but I assume that if a single-picture format doesn't have the concept of making use of reference pictures (so previous or future frames), it's leaving _a lot_ of compression potential on the table.
You can of course just have a sequence of individually compressed frames, just as you can do with PNG, TARGA, or JPEG, but these are not usually the ways you would want to distribute the video due to the huge sizes.
I'm no expert on this but I assume because it would be all keyframes? I.e. the format doesn't support encoding transitions from frame to frame. But I guess decent video support could be possible if you define video as pixel perfect animation, i.e. Animated GIF. These would of course not compete with traditional video.
Video is massive. Uncompressed 1080p video at 60fps is 1Gb per second, compared to ~5mbps for compressed video. QOI is great for pictures where 3-5x compression is about as good as you can do, but video compression is about getting close to 500x compression.