Same. I'm not really sure what it is, but the moment you add a few plugins to vim, you definitely notice a slowdown. I'm sure it's the abomination that vimL is.
I do use plug, actually. It was more a complain to the fact that the editor does feel a bit sluggish sometimes if you have a few plugins on. I think vimL is a pretty darn slow language (not to say that it's probably one of the worst languages I've ever touched).
For example if you use something like vim-plug[1] (amongst many plugin managers) then you can use on-demand loading to make keep start up times lean.