It isn't android. It's the manufacturer skins. AOSP Jellybean runs really smooth on my low end phone. (512 MB ram, 600 MHz processor) But HTC Sense 4 runs like crap.
It's a bit of both. The original Galaxy S (which was only a year and a half old at the time, and was a top-of-the-line phone) struggled really hard with stock android ICS for me.