I did this a few years ago, and as a result, I'm pretty confident I'm in the minority of people who 'like' both. I have not yet declared a side in the holy war of emacs vs. vi. They're both great.
As a text editor, I find vim's paradigm to be absolutely superb. As an editing environment, emacs offers so much more in terms of extensibility and functionality.
I tried for a long time to love readline's vi-mode, but the level of customization to get a bad vim experience (I know vi and vim are different beasts, that's part of the problem) makes this not too valuable a task. On the other hand, with emacs + evil, I get emacs in any REPL I want, and in a pretty capable shell, with evil editing everywhere, which is much less painful to configure than readline.