That’s actually a horrible model for TV. TV is often social. I would hate to have to have my phone to control my TV. Not to mention when guests come over. I’ll give someone else the remote, but not my phone.
I've never had to do that, guests get access to the playback controls while on my network and are completely free to use their own phone to cast something else to my TV if they wish. What they don't get access to is my own media library.