Put this in ~/.inputrc
set editing-mode vi set keymap vi-command
Note that this feature is broken when you have a PS1 with a \n in it, in an older version of readline/bash (I am not sure which), so if you're having this problem, just upgrade your bash version.
Put this in ~/.inputrc