If it weren't for vim, I would have definitely considered switching. I know you can map keys and all, but, it just doesn't make sense going through all the hassle.
I still can't use Colemak on iOS devices, so regular QWERTY encounters still happen, but I didn't lose much of my QWERTY speed after I switched, something I feared might happen.
Before Colemak I tried Dvorak for about a month, but uncomfortable locations of common shortcuts seemed bit unfriendly - great for typing, not as great for computer use.
I find that I don't have to actively think about what layout I am typing in, it just flows. I imagine bilinguists have a similar experience.
I chose Colemak primarily because it doesn't mess with common shortcuts (Bottom-left keys "qwarzxcvbm" remain the the same positions).