HA is extremely modular and add-ons like these tend to be API based.
For example, the whisper speech to text integration calls an API for whisper, which doesn't have to be on the same server as HA. I run HA on a Pi 4 and have whisper running in docker on my NUC-based Plex server. This does require manual configuration but isn't that hard once you understand it.
I've been using HA for years now, and I don't think there's a single feature that's not toggleable. I expect this one to be too, and also hope that LLM offloading to their cloud is part of their paid plan.
For example, the whisper speech to text integration calls an API for whisper, which doesn't have to be on the same server as HA. I run HA on a Pi 4 and have whisper running in docker on my NUC-based Plex server. This does require manual configuration but isn't that hard once you understand it.