[mmcclure]

(Mux founder here, but adding some additional community bits to the conversation)

Probably unsurprisingly, the (honestly absurd world of) color is a pretty consistent topic among video engineers. Color theory is one of those areas that people working directly with it see it as a completely unsolved problem, and everyone else largely takes it for granted. This is a small sample size, but these are all talks just from a local SF Video meetup and the Demuxed conference, and the speakers work at YouTube, Vimeo, Mux, and a fruit company.

Color (SF Video 2016) - https://www.youtube.com/watch?v=PiAiOl1Pvgk

Early Experiments in Color Vision and Their Applications to Modern Color Theory (SF Video 2017) - https://www.youtube.com/watch?v=fXd6HLqpoMk

A Jaunt Through Color Technology in Video (Demuxed 2017) - https://www.youtube.com/watch?v=XMnvY7a4-As&list=PLkyaYNWEKc...

Your browser and my browser see different colors (SF Video 2020, by the author of this post) - https://www.youtube.com/watch?v=9JXx0bao7ho

[jdashg]

I would say that much of it is actually solved in theory, just not in practice. (I.e. we know what it should display, but browsers get it wrong still) There are both outstanding issues with browsers being accidentally wrong/incomplete (e.g. both Firefox and Chrome have issues with full-range videos displaying incorrectly, just with different videos), and also with various implementations deviating from standards to "do the right thing" (though this is also a central dilemma of hdr display/tone-mapping).

There are definitely cases where it's clear what we (as browsers) are supposed to do, but some implementations have chosen to err on the side of user preference rather than correctness. I hope we can move that needle back towards "implement the specs". There's no good reason for any 601- or 709-marked content to display incorrectly in the year 2021.

Unfortunately, if you don't mark the videos, that's on you though. Fwiw Firefox generally does follow Chrome in inferring unmarked bt601/709 based on size: https://searchfox.org/mozilla-central/rev/1df3b4b4d27d413675...

Honestly I would rather issue warnings in these cases rather than silently guess.

[Daemon404]

It's been my experience that part of the reason browsers get stuff wrong even when it's clear what to do, is that entirely different teams work on parts of the browser that all need a poper color pipeline, and while one team will learn and do it correctly, the new team working on the diffrent part has to go through the same process again.

You can see this, for example in Firefox's image pipeline which seems to assume everything is 601 (because JPEG), and this means 709 AVIFs won't render correctly (and are thus off by default currently).

Or in Chrome, the MediaRecorder API will create HD H.264 streams that are untagged, and are 601. Which Chrome then assumes is 709 based off the res.

Or, also in Chrome, the WebCodecs team not having talked to the Media Team, seemingly (?) before starting work on what kind of buffer gets returned to the user, and what the semnatics of its color are. (I think this is resolved now, though - this was a year or two back when they engaged with VideoLAN over this API)

[jdashg]

Honestly a big reason is that there simply aren't tests.

[pdkl95]

I miss the days when browsers took pride in announcing[1] their new version passed Acid2/Acid3. Sure, the tests had issues, but they were self contained and anyone could see which browsers mostly passed and which were missing half of the features. Unfortunately, the modern trend towards monopoly is increasingly hostile to ideas like "open standards" and "interoperability", so I doubt we'll see this kind of test used again anytime soon.

[1] https://en.wikipedia.org/wiki/Acid2#Timeline_of_passing_appl...

[rwxsh]

The modern equivalent is the Web Platform Tests[1], which are far more extensive than the Acid tests were. Browser conformance is tracked continuously[2].

[1] https://github.com/web-platform-tests/wpt [2] https://wpt.fyi/

[pdkl95]

The WPT is the exact opposite of what I was talking about. The interesting thing about the Acid tests isn't their utility as a conformance test. Sure, that's what the were, but even at the time the problems with the tests were well known. The features they tested were incomplete and somewhat arbitrary. Most browsers didn't even bolter trying to get the last 3 points on Acid3, which tested SVG fonts. I'm sure WPT is far more useful from a technical perspective.

Unfortunately, WPT is missing the actual feature that made the Acid tests interesting: a design that could be used and understand by anyone. Most people are not going to go do the incredibly complex work required to actually run the WPT tests (which apparently involves its own command line utility to manage the process, requires knowledge of python/pip/virtualenv, and understanding platform-specific documentation for both setting up and running the tests.

For Acid2/Acid3, anybody could simply load the test URL to run the tests themselves, on their own browser and OS, and see the results first-hand. It doesn't matter that most people didn't know the various CSS/Javascript/etc features being tested. Seeing that your browser completely failed Acid3 was obvious[1]. It was exciting to see for yourself if the latest browser update scored higher on Acid3.

[1] https://en.wikipedia.org/wiki/File:Acid3ie8rc1.png

[rwxsh]

I probably shouldn't nitpick here, but Chrome's WebCodecs team is a subset of the Media team. Color simply wasn't the first priority to implement.

What has changed is that more functionality got pulled into WebCodecs as it became clear that existing web platform objects (eg. ImageBitmap) were not an ideal fit.

[rwxsh]

> Project leaders from Ffmpeg, Google, Mozilla, Microsoft (and probably Nvidia and AMD) need to all get together and decide on a single method.

Speaking non-officially as a browser implementer, VUI isn't always reliable as a source of color, both because the encoders don't always know, and the decoders don't always extract it reliably. The container is a more reliable place to put it (eg. MP4 'colr' box).

Browsers do differ in behavior when these sources of metadata differ. It currently looks likely that WebCodecs will require implementations to use app-provided color metadata over in-band metadata, which would be good for interoperability!

[Daemon404]

This is one place the ISOBMFF spec really screwed up, in my opinion. They should have included semantics for the colr box, i.e. which takes precedence.

Intead we get: "Colour information may be supplied in one or more ColourInformationBoxes placed in a VisualSampleEntry. These should be placed in order in the sample entry starting with the most accurate (and potentially the most difficult to process), in progression to the least. These are advisory and concern rendering and colour conversion, and there is no normative behaviour associated with them; a reader may choose to use the most suitable."

[Daiz]

To add to the resources here, I also ran into this issue a long time ago myself and wrote this little test tool for it some five years ago[1]:

https://daiz.github.io/yuv-to-rgb-in-html5-video/

Sad to see that even today the results don't line up with the expected... and in different ways than before, even?

Something the article doesn't touch upon is that colorspace assumptions can also vary between different video formats across browsers (at least back when I made this test), which adds another unfortunate dimension into the color accuracy mess.

Anyway, good to see more attention brought on the subject once again, though unlike the article, I would personally say that if we're going to make assumptions in the absence of concrete colorspace information, I think we should rather consistently assume BT.709 rather than BT.601, for the simple reason that we can reasonably expect that basically any camera in existence today would likely record in BT.709 by default, and I'd expect most editing software with their HD presets and whatnot to default to BT.709 as well.

[1] A related comment I wrote on HN about it back in the day: https://news.ycombinator.com/item?id=12022163

[KONAir]

(I hate having to make these statements but I am not trolling)

I was told search for "perfect" parity across users was in vain because perception of colour varied too much, from blood sugar level to physiology and etc. What is the true benefit of going after this on software side? I mean I am not understanding the difference it causes on artistic statement or emotional impact.

I understand and support "true colour" across all hard/software but is it really that impactful?

[lstamour]

We're not talking about perceived colour in these articles. Instead at best we're talking about absolute colour as might be sent over a display cable or as might be presented on a calibrated screen. Even if the screen isn't calibrated or the user is partially colour blind, the values that are sent over the HDMI or DisplayPort cable do still matter. Those values tend to be 8-bit 0-255 values for R, B and G, though that's starting to change. I won't get into how 255,255,255 might be a different colour depending on what mode your computer monitor is in or how your cable signals the mode to the computer monitor, but the point of this article is that the colours in a video can vary based on how and when the video was recorded, mastered, etc. This is partially historical: if watching a DVD you'll have a smaller colour range than if watching a UHD 4K HDR Blu-Ray because back then most TVs couldn't display high contrast images or what we now refer to as HDR or Dolby Vision. There are other concerns, of course, such as framerate and resolution and even the shape of a pixel can change in some video files. So ... it's a complicated topic. :)

[thefounder]

I doubt the blood sugar level is more important to color than different display types/models and/or different rendering software.

I think the issue is more about consistency than true color/fidelity as true fidelity/color sharing between individuals is impossible at this time. Professional video equipment exists for a reason.

[jdashg]

One of the major user stories is "I have CSS and a video on my page, and my colors should match".

Another is the desire for product photos to look relatively true-to-life, so you don't order something and it's too dark, or too red, etc.

[lattalayta]

Agreed, digital color can be a huge rabbit hole once you start understanding all of the pieces involved. Reminds me of one of my favorite xkcd comics (the alt text is particularly relevant here)

https://xkcd.com/1882/

[munk-a]

I've always found it interesting that XKCD uses a slightly beige background color in all of the comics.

[elliekelly]

Which browser are you using...?

[munk-a]

Opera version 4 on a Sun SPARCstation 5 rendered in black and white onto a twenty five year old CRT.

Why - is it not beige to some people?

[TonyTrapp]

I just took a screenshot of the site and used the color pipette, it's pixel-perfect #FFFFFF white, not beige.

[magicalhippo]

Reminded me of Colorful Notions[1], a BBC documentary from the 80s. I found it illustrated a lot of the weird quirks of colors with rather nice demonstrations.

[1]: https://archive.org/details/ColourfulNotions