If your caffeine use is interfering with your sleep, then it's adversely affecting your overall health. That's generally a good reason to discontinue or limit use of a substance.
I get it though, we all like our pleasures in life. But some of them aren't good for us, so we have to choose between the short term pleasure of substance use versus the long term pleasure of not just longer life, but higher quality life when we're older.
I started listening to podcasts/audiobooks as I went to sleep about 10 years ago due to this and it hugely decreased the time it took to fall asleep. Unusual to be more than 10 minutes now.
I get it though, we all like our pleasures in life. But some of them aren't good for us, so we have to choose between the short term pleasure of substance use versus the long term pleasure of not just longer life, but higher quality life when we're older.