Yes. Depends on client bandwidth and CPU, but it's significant. In fact, it's insanely high. Not caring about "just another 100 KB" is how people end up with 2-5 MB bundles that take seconds to download on weak networks and seconds to execute on weak devices.
You would have to epxlain how adding a 100kb library make your bundle become 2-5MB huge.
And note that with RxJS 5 you can only pick operators and data structures that you need; I have written a Redux-clone using RxJS [1] which uses Observable, Subject and a handful of operators and is less than 16kb as minified & gzipped in total as self-contained umd bundle (of which half the size is attributed to lodash helper functions).
This is news to me. Last time I tried RxJS, was unable to get a usable core less than 100 KB minified (I don't use gzip as a metric). Would consider 20-30 KB. Might want to look again.
Bundle size: a typical SPA imports multiple libraries, they import more libraries, and so on. It's not just Rx. I shouldn't have to explain how small things add up. Not caring about size is how it balloons up. We have to care about it in every library.