Throw enough caching at a speed problem, and the speed problem effectively goes away. This is why the terms "Varnish" and "fcgi_cache" are so well associated with WordPress installs.
You are absolutely correct - caching solves the apparent speed problem to the end user, not the underlying cause of the speed problem. That said, solving the apparent speed problem for the end user is frequently enough.