In my experience, it's just some particular syntaxes that vim is slow to process - large XML/HTML files in particular. But just ":synax off" when you want responsiveness more than you want colors :)
For me it’s Python code. Django views and models. Which honestly should be pretty fast because it’s all indent based but I’ve tried a number of things and while I would up with acceptable results they still aren’t lag free.