Agreed. Most of the time you end up with a smaller file as it's easier to comment out the various modules as you're going as opposed to downloading customized builds from the website as you work.
Also, if Sass/Compass is more your cup of tea, there's a well-maintained port of Bootstrap[0].
It makes the upgrade process easier if you stick to only making changes in their variables.less file[0], or even better creating a copy of it and including that instead in bootstrap.less. Heck, you can even just create your own top-level myproject.less file and copy over the includes you need using bootstrap.less[1] as a template.
Also, if Sass/Compass is more your cup of tea, there's a well-maintained port of Bootstrap[0].
[0]: https://github.com/jlong/sass-twitter-bootstrap