[cbhl]

"The solution was ... a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling ... rendered with correct camera positional data."

Gee, that sounds a lot like a holodeck. We've come a long way from using Wii Sensor Bars[0] for position tracking.

[0] https://www.youtube.com/watch?v=LC_KKxAuLQw

[modeless]

The "holodeck" version of this is called a CAVE and the first one was built in 1992: https://www.youtube.com/watch?v=aKL0urEdtPU https://en.wikipedia.org/wiki/Cave_automatic_virtual_environ...

[ci5er]

The haptics on CAVE were pretty clumsy, but boy howdy! I was pretty damn impressed when I got to interact with it at SIGGRAPH that following year.

I've got to say that the NC (Chapel Hill) pixel machine may have been a more supreme technical achievement, but it was harder to "get a feel for". As a young semiconductor geek at Motorola (Tokyo Design Ops) at the time, I did push for getting 1~4 bit CPUs on every column of 128-~192- bit deep framebuffers for a time. I almost got Sega to sponsor the project, but then something distracted me (I don't know what - but I suspect it was the Motorola 96K and a bank of AMD bit-slicer chips)) and I wandered off to do something else.

[ragebol]

For the VFX industry, the tracking had already been solved for ages, with those reflective little balls on suits etc. in a mocap system. The Wii sensor bar's thing was that it was really cheap.

But yes, damn close to a holodeck. But you can't see depth in this setup, right?

[nocut12]

If it's perspective corrected for the camera, it would probably look very distorted for anyone else on set -- whether there's depth or not

And that's certainly not the goal with this. Something along those lines has been around for a while (https://en.wikipedia.org/wiki/Cave_automatic_virtual_environ...). This system seems specifically targeted for solving challenges for film production, as it probably should be.

I am pretty impressed that real time rendering has gotten good enough to use for these purposes. I certainly wouldn't have expected that those backgrounds in the show were coming out of a video game engine.

[miohtama]

They mention they cannot push enough GPU juice to the screens, so they only render the camera focus area in full resolution. Also there is 12 frame lag which prevents moving camera too fast.

[Cthulhu_]

One solution to this would be to put the camera on a fixture that replays the same motions every time, so they can do a 'dry run' and correct the rendering. (Putting a camera on a fixture is not new, IIRC they did it in Back to the Future - https://www.youtube.com/watch?v=AtPA6nIBs5g is a really good yet succinct documentary on it)

[fhars]

IIRC they did it in Star Wars for the space battles (but it is forty years since I read about it, so may memory may be playing tricks on me).

[ygra]

They did. They only had so many models of the spaceships, so multiple takes with a programmed camera path and compositing was used to increase the number of vessels in the scene.

[toyg]

> they cannot push enough GPU juice to the screens

...yet. That's just a matter of waiting a few more years.

I wonder if most of the next Star Wars movies will be shot with this tech.

[Aeolun]

I don’t fully get this. They could just employ different computers to render different parts of their cave. It seems more like a cost savings thing than a technical limitation.

And I’m not sure why you’d skimp on a few PC’s if you’re already building a humongous led wall, so maybe there’s something I’m missing.

[GonzaloQuero]

I worked in a somewhat similar project in 2015, though not as complex as this, to build background videos for DotParty, using UE4 for panoramas and then stitching them. One of the hugest issues we found was that, because of this being a game engine, a lot of things are not deterministic, so if we used multiple cards or computers, particles and other environmental effects would not be in sync, and the stitches were glaring.

[Aeolun]

Yeah, that’s a good point. And taking out the particles and doing those separately is probably near impossible.

For the non-visible screens it wouldn’t matter that much, but they’d still end up with the moving fulcrum for the main engine.

[gmueckl]

The part you are missing is the insane complexity involved in keeping perfect frame sync with a low latency across GPUs and machines, especialliy when some final compositing of partial outputs is involved. The stuff sounds simple on paper and it looks like you can just go buy the tech, unpack it and switch it on. The reality is nothing like that. The off the shelf tech is fiddly and barely stable because it is always a low priority feature added with the least possible effort.

[Aeolun]

If you have a 12 frame delay regardless you have an awful lot of time to get your clocks in sync.

Obviously that tech is not simply unpackable, because they’re on the cutting edge. But that’s also why you could expect some customization.

[rangibaby]

“They could just...”

Pretty much every time I’ve thought this, it’s turned out I was underestimating the difficulty of doing “just” that.

If it were just that easy, wouldn’t they have done that already?

[Aeolun]

Who knows? Sometimes people make things a lot more difficult than they have to be.

That isn’t always the case, but asking the question is better than the alternative.

[RaptorJ]

From the video posted upthread, around 3:40 it looks like they're doing just that: https://youtu.be/gUnxzVOs3rk?t=220

[vernie]

The most interesting aspect to me is that the system is pulling double duty by displaying both a dynamic, perspective-correct backdrop for the camera's POV and a static view for environment lighting and reflections outside of the camera's view frustum.

I wonder if they had to take care to mitigate artifacts caused by the dynamic view bouncing off of surfaces facing away from the camera.

[BubRoss]

What artifacts?

[rangibaby]

I guess he is thinking about situations where you would use a negative fill

[mlrtime]

Look at the ceiling, you can see it moving.

[rebuilder]

Mocap systems haven't really been able to produce deliverable results without human intervention very long. I'd argue they're still not there, some filtering and cleanup of the data is usually required. A lot of VFX is still about throwing human labour at problems.

Edit: I should note I'm talking about motion capture for characters etc. Capturing the motion of a rigid object like a camera in a controlled environment is very doable.

[ChuckNorris89]

You're right. The mocap data for the actor who played Thanos in Avengers had to be post processed by artist by hand since the motion of a regular man doesn't correspond to the motion of a larger and heavier Titan. I guess in a few years all that could be automated with ML.

[estebank]

You don't need ML to solve that problem, you can use inverse kinematics:

https://youtu.be/KLjTU0yKS00

[whatshisface]

They have polarized 3D screens, and with head tracking, you have it.

[ragebol]

Was thinking the same, but that'll also work for just 1 PoV

[andybak]

You could multiplex more images if the glasses were synced and actively polarized. Every person gets a timeslice of the total image.

[jacobush]

Or VERY high framerates.

[babypuncher]

It won't trick anyone with stereo vision, but the perspective correction provides all the depth cues needed to create a convincing 3D visual for pirates and non-stereo cameras.

[lsaferite]

Interestingly I would imagine pulling the non-CG part out of the frame would be possible as they have the ability to generate the exact image without the real-world aspects. Basically a virtual green-screen. Combine that with a stereo camera and the fact that the source CG is actually 3D and you could get very convincing 3D movies I'd think. Yeah, that simplified a lot, but I'd say it's possible. And they still get the benefits of the lighting aspects as well as the immersion for the actors.

[ragebol]

pirates? What arrrr you talking about?

[frostburg]

Eyepatches.

[overcast]

Speaking of the Wii, does anyone else wish that motion controls could just be removed from its existing library? Not only has it just a major annoyance in most games, but its basically locked into that hardware now.

[mercer]

Metroid Prime is one of the few games I've played all the way through multiple times. When it became increasingly difficult to do so on a GameCube, I relented and tried the ii U version. Much as I've tried,I can't get comfortable with the controls.

Thankfully I finally have a laptop that can properly emulate the GC version and allow me to use the GC controller that I dusted off.

But I can't wait for a GC-inspired version on the Switch!

[overcast]

I've been procrastinating playing Metroid Prime 3 because I don't want to waggle around the whole time. Pumped for the endlessly delayed Switch version though, I've had it preordered FOREVER.