The trouble with armchair quarterbacking is that many things are easier said than done. It's not like they're deliberately making your code slower — the with-statement makes certain optimizations unfeasible. Their options are to always disable those optimizations or to do it only when there's a with-statement in scope — there's no third option that makes with-statements magically fast. If you can come up with new optimizations that make the with-statement faster, I doubt they'd turn down your patch.
I don't believe that this is just an oversight because "people say you shouldn't use `with`". It simply cannot be optimized in the way that normal property accessors are. If you look at old IE perf, they're actually pretty close (both really slow).