It's not at all a random website. haveibeenpwned is renowned. Your phone number is not uploaded to the server, instead your browser asks for a whole range of (hashed) phone numbers and checks locally if yours was one of them.
HIBP is indeed probably just fine, but I'm not sure how the phone number searching works. "There's no k-anonymity implementation for phone numbers at this point in time."
The process is spelled out here: https://haveibeenpwned.com/Privacy