It's not just you. I noticed it a few months ago and ended up disabling cursorline entirely. It seems to be caused by an intersection of cursorline and specific syntax highlighting rules (it happens for me in Elixir, but not in Ruby, for example). Honestly, while line highlighting is nice, I can't be bothered figuring out the way through it.
https://www.reddit.com/r/vim/comments/6onaui/vim_getting_slo...
is at least one exploration.