If in one browser an asm.js compiled Unreal Engine 4 game loads in 5 seconds, and in Safari it loads in 2 minutes, you better believe users will notice. JS performance matters.
Yes, they'd notice that. But how often does that scenario (running Unreal Engine 4 under asm.js) happen, would you say, on a day to day basis? Compared with trying to scroll a loading webpage, I mean. We're probably talking orders of magnitude difference, I suspect.