That is not the case. Only YouTube and Netflix have exceptions where the Android app on Google TV / Android TV intercepts the standard Chromecasting where the TV just pulls up the receiver web app in an ephemeral browser. And even for YouTube it'll automatically log you in in a temporary way when you cast. For anything else, casting to an Android TV device is exactly the same as casting to an old-school Chromecast.
Just to try, I just logged out of my Google Chromecast with Google TV (yes folks, it may time to rebrand...), and started playing a stream from Spotify using my phone.
And now I'm listening to Spotify.
I did this with both Google Cast and Spotify Connect (Spotify, at least on Android, allows selection of this).
No issues.
Am I not holding it right? It sure has the appearance of working correctly, but we may not be observing the same things.