[derefr]

Here I was hoping this would be something that works with the OS to take pre-window-compositor “3D screenshots” of a desktop, and then assigns the windows Z depths so that they’re floating above the desktop + each-other. Looking at the rendering of such a “3D screenshot” (in orthogonal projection) would look exactly like a regular screenshot… until you added a depth-of-field effect.

But, of course, you could also look at it in other projections; tilting it around in 3D space (as done here); applying fog to shadow “distant” windows; lighting the scene from a point-source so as to make the windows cast real shadows on one-another (with more light let through translucent areas!); etc. I would imagine that the (ideally HTML5 embeddable) viewer for this “3D screenshot” format would do all those things.

(I do hope someone does try creating such a “3D screenshot” format and viewer, as IMHO it would have a fairly-obvious use-case: reproducing static “look-around-able” snapshots of the already-depth-mapped AR window layouts in visionOS. Being able to tack arbitrary depth-maps onto windows from a 2D desktop OS would just be a bonus.)

[Pulcinella]

The closest thing I have seen to this (which isn't that close) is the UI debug viewer in Xcode. You can get an exploded-view diagram of all the UI elements that you can rotate in 3D space. No lighting or shadows though and its limited to apps you have debug access to.

I think the Amazon Fire phone also tried something similar in real time with several front facing eye tracking cameras and the gyro+accelerometer to shift the phone UI and simulate a 3D view with parallax. The old mobile Safari tab view also used to shift the tabs based on the phone's orientation.

I would love to see a "globally illuminated" UI someday, even if it's impractical. Something like all those Windows renders the Microsoft Design team put out, but in real time. Totally impractical and a poor use of electricity, but it would be cool to have path traced, soft drop-shadows.

[whywhywhywhy]

>I would love to see a "globally illuminated" UI someday, even if it's impractical

Apple patented a ton of stuff for this probably a decade ago. It seemed at some point they were going to start procedurally rendering aqua materials and the like using recipes for lighting that could all be dynamic.

Presuming some of it made it into VisionOS

[jhaenchen]

You're getting very close to the motivation behind VisionOS, aka a 3D spatially oriented operating system.