[Tomdarkness]

8k seems... optimistic. The PPI of such a display would be off the charts. Not to mention the kind of hardware required to run such a device. The NVIDIA RTX 3090 can barely run a 3D 8k game at 30fps, let alone in 3D for VR. What would you even connect such a device with? Displayport and HDMI wouldn't support two 8k streams.

[riggsdk]

Since they do mention eye-tracking they can heavily exploit foveated rendering which will make the resolution extremely high at the exact point where your eyes are looking and much lower in the outer regions of your vision. This is practically unnoticeable (if done right) and allows for much more interesting performance optimizations. Full 2x8K in 90Hz is impossible otherwise.

[jayd16]

Part of the problem with foveation is its pretty noticeable and mildly annoying for not a huge gain.

[bigiain]

Around the early/mid eighties, I got to play in a flight simulator for the new F1-11 avionics package Australia was buying (my dad ran the project building the sims).

It had a pair of Silicon Graphics Reality Engine IIs, one projecting a lower res image over the entire half-spherical screen, and the other driving a projector mounted on a gimbal that tracked the flight helmet - to display a high resolution image in the direct field of view of the pilot.

It was _possible_ if you tried, to "trick" the system so you could notice from the pilots seat what it was doing. But it was _remarkable_ the difference between sitting in the seat with the helmet on, and watching from behind where you could really obviously see the high res patch of sky moving around. Enemy planes turned from Space Invaders kind of pixel art into recognisable Russian fighter planes when the pilot looked at them. The "immersive reality" while flying the sim was amazing.

[riggsdk]

It greatly depends on the techniques used for rendering. Current rendering engines are focusing on pixel perfection at every part of the screen as they don't know where you are looking. More and more games use a hybrid between path/raytraced effects and other shader effects that could benefit from knowing where the viewer is actually looking. Especially raytracing can get huge speedups from sampling less rays: https://www.peterstefek.me/focused-render.html

Nvidia has researched in temporarally stable resoultion reduction at the edges (needed or you'll notice flickers in the blurring) as well as enhancing contrast which the eye is more sensitive to in the peripheral vision than sharp details.

Put a lot more research into this as well as proper support in the major 3D game engines and we have a winner.

[duncanawoods]

> pixel perfection at every part of the screen

Current lenses have quite a pronounced sweetspot in the centre of the vision so high resolution is wasted at the edges.

"fixed foveated rendering" is supported with Oculus and implemented directly in some games to reduce resolution at the edges just without eye-tracking so you can notice it if you move your eyes instead of your head. There is also "dynamic fixed foveated rendering" to ramp up/down for the current rendering load.

[jrrrr]

Which version(s) have you experienced? (Any that track eyes?)

I've only used Quest 1, with _fixed_ foveated, and while it's noticeable, it's good enough that I could see a generation or two of improvement pushing it beyond noticeability.

[samtimalsina]

Unnoticeable if done right.

[enw]

Thankfully, if there's any large company on Earth that really cares about perfecting UX, it's Apple.

[jayd16]

I think part of the problem is the peripheral area where you wouldn't notice but still see is so thin that you don't save a lot. Maybe Apple has cracked it, though.

[bigiain]

Might also be that peripheral vision is more sensitive to movement, and the edges where the lower res and higher res rendering meets could make for distracting discontinuities that look to the reptile brain (or the preprocessing in the retina/optic nerve/visual cortex) like a tiger...

[leecb]

>Apple will liberally use an already-known VR technique that involves using eye-tracking to render objects in the user's periphery at a lower fidelity than what the user is focusing on.

This seems to be describing foveated rendering, which is reducing the image quality in your peripheral vision, because you are less likely to notice it there. It requires tracking where the eye to so you know what part of the screen the eye is looking at.

The RTX 3090 is likely rendering the whole 8k screen at a consistent quality level, whereas foveated rendering would mean that only the part of the display that the eye is actually focused on would be rendered at full quality. If Apple could pull off the tracking well enough (accurately, with low latency), they could probably save a lot of GPU power by lowering render quality outside of what you're looking at.

https://en.wikipedia.org/wiki/Foveated_rendering

[jiggawatts]

a) Deep-learning super sampling (DLSS) allows upscaling to 8K at a low cost.

b) VR uses variable rate shading because we have a lower visual acuity in our peripheral vision.

c) VR rendering typically "shares" a significant amount of the work between the two viewports. E.g.: one set of "commands" are rendered simultaneously into two buffers with different view transforms. Textures and meshes are cached once and rendered twice, so the bandwidth requirements aren't actually doubled.

d) Display stream compression (DSC) and similar technologies would work well for VR because the viewport is always in motion with a high refresh rate. One could even imagine sending a H.265 compressed stream wirelessly at a mere gigabit, which is fantastically high bitrate video but well within current WiFi capabilities.

e) There will be future developments as well, we're not stuck with current technology. Keep in mind that current era flagship GPUs are manufactured on silicon processes that are about 3 generations old! By the time this VR kit hits the mainstream market, GPUs could be manufactured on a 3 nm TSMC process and easily put out 90fps in 8K resolution.

[shock-value]

Regarding (d), if you have a Quest / Quest 2, there is no need to merely imagine; this is how WiFi + Virtual Desktop works now for PCVR connectivity, and it’s excellent. (Not sure if it tops out at a gigabit but it’s much more than enough for a great experience at 4K of the Quest 2.)

[riggsdk]

Especially if Apple keeps pushing their custom GPU's (which are already at 3nm). The current M1 GPU is somewhere between a GTX 1050 and a 1070 - so still a few generations old. With a bit more focus on the GPU part (and use a bit more power) they might be able to pull it off.

[sbierwagen]

DP 2.0 supports two 8K streams @ 120hz with 10 bit color over USB C: https://en.wikipedia.org/wiki/DisplayPort#2.0 Requires DSC though.

[banana_giraffe]

The article mentions use of something like a M1 processor, so I'd assume this is something like the Oculus Quest where the rendering is done on the VR headset itself.

It'd also explain the price tag, if you have to buy both a high quality display and a speedy GPU that's been glued together.

[jayd16]

I wonder if the bulk of the resolution is to handle this pass through video in a fixed pipeline. Programmable rendering could be at a much lower resolution and scaled up in hardware.

[brokencode]

Like mentioned in the article, they would only be rendering what’s directly in the clearest part of the eye at full resolution, so the idea of actually needing to transmit two full 8k streams is not right.

Also, it sounds like this would be more of an all-in-one device, where it’d handle the rendering instead of connecting to a separate computer, so the rendering performance is more likely to be the limitation than any kind of transmission limitations anyways.

[pezezin]

If you want the latest AAA game in ultra quality, then of course no GPU can run it at 8k. But I'm pretty sure a RTX 3090 can run older games at 8k.

[barnabee]

I suspect an RTX 3090 could run many Oculus Quest 2 games at 8k.