Agreed. Really the only legitimate point is the increased iteration time when working with a preprocessor vs straight CSS. But even that can easily be overcome by using an auto-reloader
On a modern computer with SSD drive the iteration time is actually negligible for a not too complex stylesheet; auto-reloading might even be faster than manually refreshing your browser without a build chain.
yes, on fast computer with small sass... as the size and number of watched files increases so does the reload time, and while it's still not some huge delay on its own, it does add up when you repeat it hundreds of times a day, and can be a real pain in the ass... to the point that I usually test all my changes live in css first, and then implement the final version in sass later, for easier management.
Are you using libsass or ruby sass? I work with a 10k loc monstrosity that took 4s+ to compile in ruby and it now takes 3~400ms to compile in watch mode, with all the overhead from Grunt & autoprefixer added.
It's a separate issue to CSS processing, but do you not find that tools like live-server or browsersync that hot-reload CSS changes are more efficient than manually refreshing the browser anyway? They update quicker than a full refresh, and if you're working on interactive web apps then they have the significant advantage that any other state you've set up doesn't get discarded while you experiment with styling.
I use browsersync all the time, it's a great tool, but IMHO it's still much more hassle then just live editing css in chrome devtools and then having the browser save the changes to local css files directly. No need to reload anything, you just save it and you are free to continue working... of course, sass has many other benefits, it's a way easier to manage, but with devtools pure css workflow is nowadays really fast and streamlined, too.