Doesn't it all depend on how long they spend on the site? If it's one page and bounce, sure, that's a terrible experience. More pages than that? Now we're starting to see savings.
If you serve a rendered page of HTML with CSS links, browsers can progressively render the page as it is downloaded. Users will notice that on the first page load, particularly on higher latency connections where round trips for resources like JavaScript files are expensive.