I went this route with code-server[1]. But later decided against it because some shortcuts don't work in the browser. Also installing and managing code-server on those VMs had started becoming a chore.
I am using remote-ssh instead. Hassle-free and works like charm.
Can't comment on sudo issue because I mostly log in using root.