It was actually someone from Amazon themselves, reporting on a/b testing they did, that gave us the numbers of 100ms of latency cost them 1% in sales. People might wait for Amazon to come back, but if it's slow to navigate back and forth to compare brands, or slow to return search results, or slow to just render the page, you can easily imagine losing some percentage of people or just some percentage of what you could have sold to them.
http://news.ycombinator.com/item?id=273900