4MB is an incredible amount of Javascript. All of underscore is less than 6kb, jQuery is less than 70kb. Both of those libs fill out the "standard" library needs of most websites.
No excuse for 4mb though based on what the app does. 50k should do.
For example in a side project I tried SammyJs then realised it requires jQuery. So I just hunted for something that can route urls without the needless dependency. I chose Navigo.
Point is it takes some care but not too hard to keep the size reasonable. Also don't use npm for dependencies on a web site. But npm is ok for tooling e.g. webpack