It's not good practice to rely on a user's cache. If they visit your site twice in a span of a day, they may have already cleared their cache or pushed your files out with other sites' files.
This article demonstrates a pretty good method to reduce unnecessary byte transfer which will ultimately speed up page loading.
Caching is still a good idea, because if a user decides to browse to another page on the same site in the same visit (or refresh/post a comment), a significant fraction of the data required for it has already been loaded. If a user does come back, there's always a chance that your stuff will still be in the cache.
This article demonstrates a pretty good method to reduce unnecessary byte transfer which will ultimately speed up page loading.