If build time is a big concern, I'd suggest Brunch. It starts out fast and it caches previously built assets so that subsequent builds are incremental.
this post encapsulates why development with js can be a nightmare. you start with grunt because that's the cool one, then everyone shifts to the new cool one -gulp- but oh wait, if you are concerned about build time, you need to be using this other new hotness.