I don't know why it matters, but enforcing a JS budget turns out to rather effectively reduce the amount of problem websites down to nearly none.
Do babies go out with the bathwater? Almost certainly. But when you're running a small scale search engine, you're never going to index everything anyway, so that's entirely fine.
Why should I download 50M of Javascript implementing a buggy, bespoke browsing engine to render perhaps 1k of text when I'm using a browser that is already capable of rendering 1k of text? For example, I used to enjoy browsing the C2 wiki, but recently they decided they needed to experiment with their own Javascript based browser, thus rendering their site unreadable unless you are using the very latest web browser (that needs to be updated every 20 minutes mind you).
Do babies go out with the bathwater? Almost certainly. But when you're running a small scale search engine, you're never going to index everything anyway, so that's entirely fine.