It's straightforward to run Pound or HAProxy in front of Varnish to provide SSL-terminatation. I wouldn't consider it a wacky setup. The performance gains over Nginx for static/anonymous caching is significant.
It adds two more layers of software to configure, monitor and maintain, and my own tests disagree on the performance gains for a fairly typical Wordpress install.
The response times were virtually identical, and the load on the box was significantly lower with Nginx over varnish, so the simplicity was the deciding factor.
The response times were virtually identical, and the load on the box was significantly lower with Nginx over varnish, so the simplicity was the deciding factor.