If the pdfs were served by some of the megacaps who posts tons of papers (e.g. Google or Facebook) then it would be an order of magnitude faster. And said megacaps would end up spending peanuts relative to the value they get from arXiv.
An order of magnitude faster sounds unimportant. Is all the time spent by researchers sitting around waiting for 0.4s for a paper to download really an issue?
They probably could make it faster by rendering server-side a preview of sorts. I don't think that most papers are large enough for this to have a major impact unless you have a very slow connection.