After you add all that, what are you using Varnish for? Cache? Placing your cache at the load balancing instead of fetching pages from the network does actually save processing and memory.
If you use Varnish, it's almost certainly sitting at the best point for doing TLS termination.
If you use Varnish, it's almost certainly sitting at the best point for doing TLS termination.