Are you using a remote VM or are you running it local? I'm using a decent MacBook Pro (2014) with 8GB of RAM, but running an Ubuntu with VirtualBox is performant enough to work in it. Or are you just ssh'ing into your VM ?
Running it in a local virtualbox VM. I have a 4-core imac with 16 GB of ram and throw all 4 cores at it and a good portion of the RAM. Going with a fixed disk size instead of dynamic gave a notable performance increase, too.