[IvoDankolov]

Now implement a raytracer that infers lighting location and reflectance (and subsurface scattering) of the objects in the scene from the image and then we can have a whole new level of realism.

In all seriousness, though, I do wonder what kind of processing power you would need in a handheld device to be able to do that. Could we realistically achieve it within 15-20 years? It's certainly one of those "gimmicks" with extreme potential.

That little tangent about realism aside, the engine itself does look quite remarkable in how smooth it is able to run. I wonder hwo well it handles occlusion, changes in lighting and the other benchmarks for computer vision, as it was not demonstrated at all in the video. In fact, the presenter quite handily avoided putting his hand between the camera and the soda can.

Other than that, I can't say much without trying the thing, but I do not own an iOS device and don't plan to in the near future. If someone decides to try out the framework, I'd be glad to read a more detailed analysis of it.

[zeen]

Fully detecting a scene's lighting is a complex problem. This paper from last year was quite interesting: "Rendering Synthetic Objects into Legacy Photographs" - http://kevinkarsch.com/publications/sa11.html

This combined with existing techniques of constructing 3D geometry out of videos could work really well if say you wish to construct a game level out of a real scene.

Raytracing complex scenes on a hand-held device at interactive frame rates is still quite a bit off. But techniques where a lot of the calculations can be done offline are possible to use today (radiosity, etc).

[rmc]

we can have a whole new level of realism.

What would you be able to do that you can't do now?

[IvoDankolov]

As Geee stated, it's about imitating the lighting of the scene. There's all sorts of subtle and not-so-subtle effects that come as a result of light bouncing around.

While it may be obvious when a virtual object is missing a shadow (as in the demonstration video), even implementing that would not be enough to fool the human brain completely, though it isn't always obvious what the problem is, only a subconscious nagging.

If you're interested in what precisely the effects that the real world has and augmented reality generally doesn't have - i'd say the biggest ones are shadows (including soft shadows [0]), depth of field [1], ambient occlusion [2], and indirect lighting.

You don't necessarily have to write a full-blown raytracer with global illuminasion, casting billions of rays to get a passable result. All of the above mentioned things can more or less be approximated in some way, and most modern game engines do so (I don't know of any phone games that do, mind, as the calculations are still non-trivial, but I do remember an interesting tech demo from Nvidia).

The biggest problem, though, is that we can do these things in a controlled setting, where you know the exact shape, texture and light reflective properties on every object in the scene. As you can imagine, that is not so in the case of augmented reality (as I quipped in my earlier post, you would have to infer surface reflectance from the image in some way). Compared to the problem of global illumination in a virtual scene - well, let's just say it's orders of magnitude harder. I don't recall anyone actually having scanned a scene with a single camera. And that's an AI breakthrough that indeed people would talk about.

[0] : http://en.wikipedia.org/wiki/Soft_shadows [1] : http://en.wikipedia.org/wiki/Depth_of_field [2] : http://en.wikipedia.org/wiki/Ambient_occlusion [*} : Also, google images

[Geee]

You would be able recreate the lighting conditions of the scene. This is the bottleneck of every AR application that aims for realism. Basically, you'll have to recreate every light source, every object, surface and volumetric properties of all objects to render completely realistic 3D objects in the scene.