That is the biggest threat - and likely where things will end up eventually… it’s when that “eventually” is and what the server based providers can pivot to in that time.
This will probably happen unless the industry conspires to roll back the availability of general computation so common people can only own computers with enough power to be glorified thin clients. The way this might look is good hardware never officially being banned, just priced too high for anybody to afford, and produced in small quantities to keep it that way while all production shifts to making massively expensive powerful hardware for corporate buyers.
Seems unlikely. We're already seeing specialized hardware optimized for LLM performance (taalas, groq, cerebras), and simple economies of scale result in these sorts of products being a better value when rented from a server vs purchased/managed/upgraded for the typical the user.
Frontier models will continue to be either exclusively available from servers or significantly more affordable from servers vs local alternatives for the foreseeable future.