Good spot! That is the product working as intended though. The background doesn't exist except as an asset that replaces the green screen. The tool is meant to replace the green screen without the need for manual rotoscoping. Even in a traditional process, the distortion needs to be done by VFX as a separate process. To do that though, they still need the green screen keyed out and this tool does that.
True, but with visual art there is what is correct and what looks correct. When things are moving and the area small no one is going to notice.
But now that is problem is solved a director will come along and say... I want a scene with a big glass of water and the camera will zoom in on it and will see the monster refracted through the glass.
I wouldn't call it a bug. This is a first step, not a final step. Maintaining the refraction might be more realistic, but it's not necessarily what the creator wants.
When you watch the video it becomes pretty clear why it wouldn't be able to do that, although it's fun to think about how a future iteration or alternative might be able to credibly (if you don't look too hard) mimic that someday.
>there’s no way that’s going to happen automatically
They train their model in a pretty straightforward way, it can also be used to capture the distortion as well, just use a non-monochrome (possibly moving) background optimized for this. It's a matter of effort and attention to detail during training (uneven green screen lighting, reflections, etc), not fundamental impossibility
Yes. But the main issue is in the way they formulate the problem. Their output is always a transparency mask, which of course will never handle distortions.
Right. Things like this are why it’s difficult integrating AI into professional movie pipelines— they’re super complex in ways AI cannot (yet) replicate for very good reasons that seem superfluous or trivially replaceable by people not familiar with them.
People in ML have this kind of belief rooted in the bitter lesson, that everything will eventually sort itself out given enough scale and data. That often makes them ignore the nuances of particular problem domains. CC is the opposite of that, it's just impossible to do everything at once.
It’s certainly a big part of the ML scene, but to a slightly lesser extent, a cultural facet of development in general. It’s not all bad! Many people have solved problems that nobody in their right mind would have attempted knowing the nitty gritty details; often the problem they solved wasn’t the one they intended to solve, or they only solve one small subset of it, but were still valuable advancements. Unfortunately, that also leads to reinforcing some people’s Dunning-Krueger-fueled insistence that they can solve another field’s difficult problems with a few thought experiments, and the only reason it hasn’t already been solved is because nobody thought to ask a developer as smart as them to momentarily consider the problem. Non-developers in tech often bear the brunt of it: moving into design after a decade of dev work, that irritating mindset was one of the reasons I left tech altogether a couple years later.
youd have to train it to also generate and st map of the distortions but creating the ground truth version of that from the synthetic data would add a lot more to render. also its very easy to plausibly fake, its not something humans are good at seeing and knowing its wrong. you can tell its completely missing but accurate vs just distorted in a plausible way is not something most brains are tuned to notice.
Not a bug, creating distortion in the comped in backgrounds is not what this tool does. It creates a transparency mask. How do you propose a transparency mask captures distortion artifacts?
That distortion to the new background would have to be added in by the artist.