[jheriko]

this is interesting (although this sort of effect has never mystified me) - i can't help but think how this can be optimised. the steps gone through here are things i never would have done and the final solution is about where i would start...

just blindly tracing to all of the verts 'feels wrong' its the sort of thing that is intuitively unreasonable with large data sets and is quite obviously going to be constrained by available horsepower at some size.

precomputed vis data is the obvious optimisation and removes the O(n) behaviour. not only do you not have to test everypoint but your raycast itself need not test every edge. i wrote something ages ago about generating PVS in 2d as an example of how to approach the problem in 3D...

(http://jheriko-rtw.blogspot.co.uk/2011/05/visibility-determi...)

the other thing that bugs me here is the jitter. it is the classic 'lazy' way to soften shadows... a better effect can be gotten for less - if you offset the light position perpendicular to the direction toward the test vert, in both directions by the same amount then you can generate two ray casts. triangulating the resulting mesh might be a little painful (unless you cut the world into convex chunks - then it is nearly trivial) but you will get the outline for both edges of the shadow and can then do something like vertex interpolated colour to give you the same visual result as infinite raycasts for the price of two...