Jetpack has some nice features, the comments form especially. It is, however, very bloated - especially since you'll often only use 2 or 3 features on any one site.
Since Jetpack code is also what we run on WP.com (tens of billions of pageviews) it goes through a huge amount of performance tuning and optimization. The way modules work when you turn them off they don't have any overhead, similar to turning off a plugin. If you were using literally one thing it might feel like a lot, but as soon as you use 2-3+ things Jetpack does it's a lot more efficient than separate individual plugins to accomplish the same task.