I tried org-mode for a while, but found it overly complicated. I use Emacs heavily, but developed a couple of Pandoc extensions which give me the org-mode features I need without all of the baggage: http://chriswarbo.net/essays/activecode/index.html
You can use org-mode with Vim more or less the same as you'd use Markdown if you're planning to compile your doc with Pandoc. You can even use README.org on github in place of README.md, and it will be turned into HTML automagically the same way as Markdown.
Why use org-mode as a format instead of Markdown? Internal cross-references. Org-mode feels just as light weight to me as Markdown for the most part, but it has fewer cases where you can't make things work. Worst case in org-mode, you mix in LaTeX.