If you're writing plain C#, Neovim with the Rosyln LSP is pretty solid. If you're working with Razor Pages, though, may God have mercy on your soul. I can't even find good syntax highlighting, never mind all the other functionality I expect out of a language server.
Visual studio is the only real alternative, for full featured IDE, yes. It is about as useful as Rider imo, but the analyzers and general UX are lacking
Not sure about Vim, but Neovim would likely yield a better experience. I use Emacs and the Vi family editors myself as well as Rider, VS, and VS Code. As painful as it is for me to admit, Rider/VS are hard to beat.