from my understanding, S3 optimizes throughput but not latency. I'm not surprised that his performance serving from S3 is perfectly fine, but I think the "answer" is to skip straight to CloudFront. CloudFront can mirror directly from your appserver, or from S3, it doesn't matter.
update: did a little googling, and found only anecdotal evidence that latency to S3 is fine but not awesome: