Unless you're planning on inlining your CSS in the head tag, IMO the network difference between 7kb and 20kb of CSS is just splitting hairs. I've gotten far more out of optimizing my CSS for programmer understanding than network size (though the two tend to be correlated!).
As someone who sees CSS as a necessary evil, the simplicity and readability of Milligram really appealed to me. I tend to use Bootstrap out of habit (plus there are plenty of resources and answers out there) but as my skills improve I really can see the advantages of a minimal, tidy starting point that Milligram (and others) provides