Ultimately it's all because of ads. If we all had to pay for our news, then we'd be more cautious and reluctant about clickbait headlines. And media would not be in a race to the bottom, affecting news quality.
Reason clickbait works is because it is needed to drive page views. In a subscription model, media sites would have an incentive to put out stories that are interesting beyond the first page view.
And even if clickbait is banned and subscription mandatory, media would be in a race to the bottom. Just a slightly different one.