From what it looks like to me, it's not so much about a drastic new feature set but rather about trimming down dramatically. You could argue that it really should be like 3.2 -> 3.3 similar to OSX Leopard -> OSX Snow Leopard. I think they gave it a full version upgrade because it WILL (not might) break a lot of apps. That's my interpretation. I might be wrong.
Aaron Patterson (tenderlove) gave a presentation where he said he wanted to refactor rack to make it more simple. He said Rails 3 had a lot more depth in the stack and it was making it slower. (when i start up an old rails 2 app i can't believe how fast it is). I don't think that is in Rails 4 though. I guess that is a separate project.
Also deprecating vendor/plugins means Heroku will have to change the way they deploy. You can notice that by the deprecation messages when you start a heroku console.
https://github.com/rails/rails/blob/master/guides/source/4_0...