[nopenopenopeno]

I am interested in using video compression to improve delivery of VR and AR content over the web. But, this can mean a lot of different things! I know there is a lot of opportunity for this, I have a lot of ideas, but my knowledge of video compression and video streaming is limited. There are some companies working on this, but it’s not a singular problem or solution. I expect it will end up being many categories of solutions.

I usually like to learn about things from first principles so that I can then choose what to intentionally skip over, if that makes sense. I find the history is usually a good place to start, but it only goes so far with these more blackbox type topics like video compression.

[dylan604]

There's a bunch of details that can be suggested, but the sad (actually, amazing) thing is that compression and encoding software has come such a long long way, that most of the default settings will produce a very decent result. FFMPEG + x264/x265 are stunning with minimum inputs required (minimum compared to the many many switches you can fiddle with individually).

For a broader understanding, look into the basics of how codecs like h.264/h.265 work. Learn about I frame encoding and how P and B frames work. Learn about the affects of long-GOP (multiple seconds) encodes for streaming purposes vs shorter GOPs (half second). Understand how an encoder makes it's vector decisions. How/why static videos encode compared to fast action. How will the videos you intend to use for VR/AR fall into those categories.

Resources like articles on Wikipedia have lots of information on the technical side of things without that are not video compression tutorials. However, learning the tech side will help make informed decisions on what knobs to tweak and why. There are so so many actual blogs/tutorials/SO answers/etc that will give you actual settings to apply. Once you have the specific question, I will be amazed if there is no answer available for it online.

[astrange]

It's all made of inter prediction + intra prediction + residual coding.

There's an evolution from JPEG -> MPEG2 -> MPEG4 part 2 (XviD) -> AVC -> HEVC… where it gets more complicated over time but that's still the structure.

There's also alternative ideas like wavelets (JPEG2000) and many other minor silly ideas; almost all of them are bad and can be ignored. Which is not to say the MPEG codecs are perfect.

ML people think they can do "ML video compression" which will be a black box; I think this might work but only because calling your video codec ML means you can make your codec 1TB instead of a few hundred KB.

[nopenopenopeno]

All noted. I’ll look up the items I don’t know. Thanks.

And yeah, I didn’t mean blackbox like ML. I meant like H.264/H.265. Blackbox isn’t the right term.

[dylan604]

There are many writeup/blogs about people's experience using these codecs, and typically, I roll my eyes at just another releasing the same info. However, for someone like you, being one of the lucky 10000 TIL types, you're their target audience.

The best way to learn is to just keep encoding. Take a source file and make an output that looks good. Compare things like its filesize, macroblocking, mosquito noise levels, etc. There are PSNR and other similar type tests that will compare your output back to the original. See if you can then tweak any of the settings to improve the PSNR score without increasing the bit rate. Then, keep decreasing the bitrate to see what things you can get away with before it becomes unacceptable. You can spend a lot of time going frame by frame comparisons, but remember, 99.99999% of viewers will ever only see it in full speed playback, so don't forget to take that into consideration. Look for obvious banding from gradients. Does a 10bit encode vs 8bit improve that? Is it worth the limits from some players not being able to use 10bit? How does frame size vs bitrate affect the quality of your file?

Doing enough of these tests, you'll start to get specific questions. Those will have more easily found answers.