For zsh, just enable vi input mode? (bindkey -v in .zshrc) It starts in insert mode, hit esc to get to normal mode where your typical vi binds work. So instead of ^a to go to the beginning of the line it's <Esc>0
I think the issue that I had was that my regular bindings in insert mode (CTRL-a) were disabled while in vi input mode, which caused problems for other shell programs. Plus, with Tmux and GNU Screen already re-configured with vi bindings, managing key binding collisions became complicated with vi input mode enabled. The bigger issue that I tend to have is managing copy-paste buffers between Screen, Tmux, and Vim.