This is completely true, but in my opinion there's no overlap of usecases where either one of these frameworks or vanilla js are both reasonable solutions. If you have a usecase that vanilla js is reasonable for, using a framework in the first place was just a plain bad choice, most likely.
I'm not sure everyone makes that consideration. I reckon many developers would chuck a JQuery or React dependency in without thinking about whether it is the right thing to do, because they are used to it.
Doing things in vanilla js just means I end up duplicating a lot of the framework code. Vue really was very simple and minimalist. Just pass it your object and it's reactive and very easy to get started. Fairly unobtrusive as well.