The difference is consent.
Google is obeying robots.txt. Google caches a copy of what you serve and gives that out. That is part of the deal.
Nobody does that. The comparison doesn’t hold.
The difference is consent.