[bglazer]

I was recently reading the section on importance sampling in David MacKay's "Information Theory, Learning, and Inference Algorithms". Page 373-376 in the linked pdf (http://www.inference.org.uk/itprnn/book.pdf)

He shows that importance sampling will likely fail in high dimensions precisely because samples from a high dimensional Gaussian can be very different than those from a uniform distribution on the unit sphere.

Consider the ratio between a sample at the same point from a 1000D Gaussian and a 1000D uniform distribution over a sphere. If you sample enough times, then the median ratio and the largest ratio will be different by a factor of 10^19. Basically, most samples from the Gaussian will be fairly similar to the uniform. A few will be wildly different.

Perhaps I'm misunderstanding both the post and MacKay's book. I'd be happy to be corrected.

[kgwgk]

If you sample from a 1000D Gaussian, most of the points will be "close" to the hypersphere of radius sqrt(1000). The distance to the center for 99.5% of the points is between 31.6-2 and 31.6+2. 99.997% will be between 31.6-3 and 31.6+3, 99.999997% will be between 31.6-4 and 31.6+4, etc.

This is what he means when he says "practically indistinguishable from uniform distributions on the [unit] sphere." As tgb remarked in another comment, the "unit" bit is incorrect.