Unless you have a very specific need, consider switching to a web server more explicitly designed for static content. Nginx takes minutes to install and configure.
from my understanding, S3 optimizes throughput but not latency. I'm not surprised that his performance serving from S3 is perfectly fine, but I think the "answer" is to skip straight to CloudFront. CloudFront can mirror directly from your appserver, or from S3, it doesn't matter.
update: did a little googling, and found only anecdotal evidence that latency to S3 is fine but not awesome:
I hope you don't mind, but I just mirrored it on Github (https://github.com/seancron/kathack). You can use the following javascript to load it from the mirror.