Keep in mind that standards are moving slow, CODEC standards more so. The golden standard is still h264/AVC, which dates back to the nineties. This is primarily due to many appliances (set top boxes, cameras, phones, TVs) using the absolute cheapest hardware stack they can get their hands on.
Compared to other standards in streaming media, I'd say that AOMedia has found adoption a lot quicker. h265 (HEVC) was all but DoA until years after it's introduction Apple finally decided to embrace it. It is still by no means ubiquitous, mostly due to patent licensing, which significantly drives up the price of hardware in single digit dollars price range.
Anecdotally, consider that Apple's HTTP Live Streaming protocol (till version 6) relied on MPEG2TS, even though Apple lay the groundwork for ISO/IEC 14496-12 Base Media File Format aka MP4. The reason was that chips in the initial Iphones had only support for h264 using mpeg2 transport streams, and even mp4 -> mp2 transmuxing was considered too resource intensive.
> h265 (HEVC) was all but DoA until years after it's introduction Apple finally decided to embrace it
No? You're talking in terms of PC / phone hardware support only. HEVC was first released June 7, 2013. The UHD Blu-ray standard was released less than 3 years later on February 14, 2016 - and it was obvious to everyone in the intervening years that UHD Blu-ray would use HEVC because it needed support for 4k and HDR, both of which HEVC was specifically designed to be compatible with. (Wikipedia says licensing for UHD Blu-ray on the basis of a released spec began in mid 2015.)
>Anecdotally, consider that Apple's HTTP Live Streaming protocol (till version 6) relied on MPEG2TS
This sounds like you might be confusing that MPEG2TS might have something to do with the video encoding instead of it solely being the way the video/audio elementary streams are wrapped together into a single contained format. The transport stream was designed specifically for an unreliable streaming type of delivery vs a steady consistent type of source like reading from a dis[c|k]. There is nothing wrong with using a TS stream for HLS that makes it inferior.
> There is nothing wrong with using a TS stream for HLS that makes it inferior.
Not wrong, but a bit surprising. As you mention, transport streams are designed to operate over unreliable connections (like satellite or terrestrial transmission). Reliability is not an issue with with HTTP or TCP.
Other than being archaic, some disadvantages are that TS has somewhat more overhead than (f)MP4 and poor ergonomics for random access / on the fly repackaging caused by continuity counter, padding, PAT/PMT resubmission timing, PCR clock.
If it were up to me to design a streaming protocol like DASH, HLS, Flash, or SmoothStreaming I'd instantly choose mp4 (or plain elementary streams). I wouldn't even consider TS or PS unless some spec forced me to.
>Reliability is not an issue with with HTTP or TCP.
We seem to be confusing what reliable means here. Yes, HTTP/TCP can reliably transmit data in the fact that if packets are missed they will be resent so that you can be assured that the data will eventually be delivered. However, that doesn't do well for real time streaming of data that is necessary to be received in order. That's why UDP was made.
>If it were up to me to design a streaming protocol like DASH, HLS, Flash, or SmoothStreaming I'd instantly choose mp4 (or plain elementary streams). I wouldn't even consider TS or PS unless some spec forced me to.
Well, it's a good thing we didn't have to wait for you to come around and design a streaming protocol and we've been able to use it for the past ~20 years with the technology that was available at the time. Perfect is the enemy of progress.
What I meant to point out as odd about Roger Panthos and co.'s decision to build HLS on top of Transport Stream containers is that Apple had already laid the foundation for MP4 with QuickTime.
Since HTTP live streaming was never about anything but HTTP, container capabilities like auto-synchronization offered by mpeg2TS were moot. It would therefore seem logical for Apple to build HLS upon what they already had with QuickTime + iso2 fragments. That was more or less the route Adobe/Macromedia had taken with Flash streaming.
Yet, the choose mpeg2TS (initially only muxing AAC and h264). The reason, historically seems to have been driven primarily by the capabilities of the iphone hardware which supported this out of the box! Separate transport streams for audio and video, WebVTT, elementary stream audio were added much later, and fragmented MP4 was introduced only once HEVC was bolted on.
I'm all for favouring what exists over what's perfect; it's just odd that Apple choose to (initially, at least) regress to 90s technology while rest of world had already adopted superior container.
> Compared to other standards in streaming media, I'd say that AOMedia has found adoption a lot quicker. h265 (HEVC) was all but DoA until years after it's introduction Apple finally decided to embrace it.
Was it all but dead because people thought h264 was good enough, until 2.5 and 4k became more prominent in media consumption? It seems really useful if youre doing encoding at resolutions than 1080p, and it makes me less regretful that I have a bunch of recent hardware that didn’t get av1 hardware support :)
> Was it all but dead because people thought h264 was good enough, until 2.5 and 4k became more prominent in media consumption?
(Lack of) a compelling use case certainly played a role. Sure, reducing bandwidth is in itself a noble (and potentially profitable) goal but why fix it if it ain't broken? Ie.: the h264 infrastructure for HD was already there, working fine.
Another factor was that HEVC was full of new patents and the patent pool licensing costs hadn't yet settled.
memcpy is ±99%. The 1% involves bit shifting (nal unit reshuffling, avc3/avc1 fixups).
So indeed, repackaging mp4 <-> mp2 containers is pretty trivial. Nevertheless, Apple initially choose mpeg2TS because it conveniently allowed them to shove the reassembled media segments straight into the dedicated AV chip.
The H265 patent licensing situation is famously a mess and has been a big barrier to adoption. Except in circles where people worry less about that sort of thing: Warez, China, ...
The licensing shenanigans of H265 was a big motivator for creating AV1, a royalty free codec.
The person who benefits from a more efficient codec tends to be netflix/youtube (lower bandwidth costs), and they are far far removed from the chipmaker - market forces get very weak at that distance.
People never stopped using VP8. In fact, your screen sharing is probably wasting excessive amounts of CPU every day because there is no hardware support.
AV1 isn't particularly behind schedule compared with previous codec generstions. We could and should have moved faster if everything went well but Qualcomm in particular were being awkward about IP issues.
Luckily the effort behind the David software codecs kept up the rollout momentum.
The AV1 ASIC takes up space on the SoC, so effectively it decreases the performance of other parts. This could be why some manufacturers have delayed including support for quite a while. Though Mediatek already had AV1 support three years ago.
Especially when Google and AOM promise to have a hardware encoder and decoder to be given out for free by 2018, and be implemented in many SoC by 2019, wide availability support and AV2 by 2020.
Well the basic answer is that, making an efficient hardware encoder and decoder, within power budget and die space, all while conforming to standard because you wont have much of a chance to correct it, and implementing it into the SoC design cycle which is and always has been at least three years minimum, is a lot harder than most software engineer at Google and AOM would thought.
Apple has a tiny sliver of the patents in HEVC, and while we don't have the numbers I feel pretty certain they pay far more into the pool to ship HEVC in their devices than they get out of it. The same is doubly true of Qualcomm who aren't even a part of the pool.
HEVC was finalized in 2013. AV1 was finalized in 2018, and has just finally started getting a robust ecosystem of software and hardware.
It wasn't really all that slow in general, just slow on dedicated streaming hardware.
Basically, it was the push to 4K (and especially HDR) that caused HEVC to roll-out. In 2016 4K Blu-rays started coming out and they were all HEVC 10-bit encoded. It took a couple more years before dedicated streaming devices and lower-end smart TVs bothered to start including HEVC support as standard because at first 4K content was uncommon and the hardware came at a premium.
Now that it's mostly the de-facto standard, we see HEVC support in basically all streaming devices and smart TVs.
AV1 didn't have any sort of resolution change or video standard change to help push it out the way HEVC did, so it's basically rolling out as the parts get cheaper due to pressure from streaming giants like Google and Netflix rather than due to a bottom-up market demand for 4K support.
I didn't know about Blu-Ray being relatively prompt. But I still think HEVC adoption was slow in broadcast TV, which I would have thought was a shoo-in market.
Qualcomm isn't even a part of the HEVC alliance patent pool, so that theory doesn't hold. Indeed, the fact that Qualcomm is currently building AV1 support into their next chip (purportedly) puts them at risk of being sued because while AV1 is open, we all know how patents work and there are almost certainly actionable overlaps with the pool.
Apple ships probably more devices than anyone, and given that the patent pool is huge as mentioned odds are overwhelmingly that it costs them money to support HEVC / HEIC, not the reverse. That theory also is dubious.
Remember when everyone was yammering for VP8 support? Then it was VP9 support? Now it's AV1. Sometimes it takes a while to shake out. By all appearances AV1 is a winner, hence why it's finally getting support.
>Apple ships probably more devices than anyone, and given that the patent pool is huge as mentioned odds are overwhelmingly that it costs them money to support HEVC / HEIC, not the reverse. That theory also is dubious.
This is a nit that doesn't negate your main point: Apple may ship more complete devices than anyone, but Qualcomm makes up significantly more of the SoC manufacturer market share[1] at 29% vs Apple's 19%
Netflix (and Youtube? I forget) will push an AV1 stream if you have the support. This was even mentioned in Apple's show yesterday. So the egg is already there and the chicken is slowly coming, thankfully.
YouTube was the first to support it. They even went to war with Roku over it and Roku killed the YouTube TV app in retaliation to YouTube's mandate that all next-gen devices support AV1, so YouTube went ahead and embedded it inside the regular YouTube app.
Roku's latest devices to support AV1, so I guess either the price came down, they struck a deal, or Roku just lost to the market pressure after Netflix pushed for AV1 as well.
I think a lot of content creators really want AV1 because of the drastic reduction of file sizes. Streaming companies want it to catch on because of the drastic reduction in bandwidth use.
I thought Google was the main one behind AV1. Couldn’t they use their position as one of the world’s biggest video platforms to break that chicken egg loop?
They have. They literally threatened to pull their support from devices if they don't implement the codec in hardware. Roku's spat with Google was a big-ish story when that happened.
I don't know how that can be viewed as a good thing.