[njoyablpnting]

Very well-executed version of this. I think this is the right interface for video editing going into the future.

I've spent a bit of time on something related, AI-generating motion graphics videos from code, also editable/renderable in-browser. Here's a few things I ran into:

- I see you mentioned being aware of Remotion in another comment, in my experience Remotion is not the right tool for adding motion graphics to what you're building. There's a few reasons for this, but basically declarative markup is not a great language for motion graphics beyond anything very basic. Also, in-browser rendering is only going to work with canvas-based components. I also wasn't a huge fan of their license.

- WebCodecs may not be as reliable as you think. I've verified several issues where I get a different output across browsers and operating systems, and even different permutations of flags, browser and OS. Is there a reason why your tool needs to be browser-based?

[ishandeveloper]

- On Remotion, yeah, not sure it's the right fit, but honestly the sheer capability of models at writing code these days has surprised me. Funnily enough, this is how I used to make small graphics for videos 2-3 years back when I knew nothing about After Effects.

We've been eager to experiment with this for a while, just have to prioritize other user requests for now. Will definitely try a few approaches and see what sticks. (Also noticed they have an experimental client-side rendering version built on mediabunny, haven't tried it yet: https://www.remotion.dev/docs/client-side-rendering/)

- On WebCodecs, there are a fair set of challenges, but we wanted to take the bet. The reason we're browser-based is the same reason I love Figma and Google Docs: no install, no waiting, just open and start. That said, for broader codec support (ProRes, RAW, etc.) we'll rely on server-side transcoding with proxies where needed.

[njoyablpnting]

> On Remotion, yeah, not sure it's the right fit, but honestly the sheer capability of models at writing code these days has surprised me.

Just to clarify I still think code-driven graphics is the correct approach, but in my case I opted for a different library with a more powerful imperative API.

> Also noticed they have an experimental client-side rendering version built on mediabunny

Yes, I've tried it out, it was a non-starter for me because it only supports canvas-based components, and Remotion didn't seem to have good support for text on canvas because they rely on HTML for most of that.

> On WebCodecs, there are a fair set of challenges, but we wanted to take the bet

Totally understand the appeal and immediacy of a browser app, I was lured in by that too. For what it's worth I've reported showstopping WebCodecs issues in Chromium and there's basically no indication they'll get fixed on a predictable timeline.

Another issue I ran into that I just remembered is animating text on canvas. It's basically impossible to get pixel-perfect anti-aliased text animation using a canvas. I would have to dig up the exact details but it was something to do with how browsers handle sub-pixel positioning for canvas text, so there was always some jitter when animating. This coupled with the aforementioned WebCodecs issues led me to conclude that professional-quality video rendering is not currently possible in the browser environment. Aliasing, jitter and artifacts are immediately perceptible and are the type of thing that users have zero tolerance for (speaking from experience).

This is not meant to be discouraging in any way, I've just been very deep into this rabbithole and there are some very nasty well-hidden pitfalls.

[spuzvabob]

> Totally understand the appeal and immediacy of a browser app, I was lured in by that too. For what it's worth I've reported showstopping WebCodecs issues in Chromium and there's basically no indication they'll get fixed on a predictable timeline.

Interestingly I have the exact opposite experience, I've reported issues both in the WebCodecs specification and the Chromium implementation, in all cases they were fixed within weeks. Simply though reports on public bug trackers and it wasn't really a major issue in any instance.

> Another issue I ran into that I just remembered is animating text on canvas. It's basically impossible to get pixel-perfect anti-aliased text animation using a canvas. I would have to dig up the exact details but it was something to do with how browsers handle sub-pixel positioning for canvas text, so there was always some jitter when animating. This coupled with the aforementioned WebCodecs issues led me to conclude that professional-quality video rendering is not currently possible in the browser environment. Aliasing, jitter and artifacts are immediately perceptible and are the type of thing that users have zero tolerance for (speaking from experience).

We're doing SOTA quality video rendering with WebCodecs + Chromium with millions of videos produced daily, or near SOTA if you consider subpixel AA a requirement for text. In general for pixel perfection of text, especially across different browsers and operating systems, you can't just use text elements in DOM or in canvas context, instead text needs to be rasterized to vector shapes and rendered as such. Honestly not sure about potential jittering when animating text, but we've never had any complaints about anything regarding text animations and users are very often comparing our video exports with videos produced in Adobe AE or similar.

[njoyablpnting]

> Interestingly I have the exact opposite experience, I've reported issues both in the WebCodecs specification and the Chromium implementation, in all cases they were fixed within weeks. Simply though reports on public bug trackers and it wasn't really a major issue in any instance.

That's fair, they are responsive most of the time. I do have one major rendering issue in particular I've been waiting on with no movement for months, so I might be biased.

> We're doing SOTA quality video rendering with WebCodecs + Chromium with millions of videos produced daily, or near SOTA if you consider subpixel AA a requirement for text. In general for pixel perfection of text, especially across different browsers and operating systems, you can't just use text elements in DOM or in canvas context, instead text needs to be rasterized to vector shapes and rendered as such. Honestly not sure about potential jittering when animating text, but we've never had any complaints about anything regarding text animations and users are very often comparing our video exports with videos produced in Adobe AE or similar.

So you use a library that takes in text and vectorizes it to canvas shapes? That could work in theory, do you have a demo of this?

[spuzvabob]

> So you use a library that takes in text and vectorizes it to canvas shapes? That could work in theory, do you have a demo of this?

Yea, it's harfbuzz compiled to WASM: https://harfbuzz.github.io/harfbuzzjs/ Then all text layout features must be implemented on top of it, like linebreaking, text align, line spacing, kerning, text direction, decoration etc.

[dandaka]

> in my case I opted for a different library with a more powerful imperative API.

would you mind sharing the name?

[njoyablpnting]

I'm using my own fork of https://github.com/motion-canvas/motion-canvas

It's not really designed for the animation code to be dynamically changed on the fly, but I've hacked together this feature in my fork.