I would assume from a UX perspective, the movement from handset phone back to smartphone screen is seen as an inconvenience and interrupts the flow of the process. Add in that many phones have screen conditions tuned to what context the phone is being used, you increase the chance of interrupting the process when the mechanisms within the phone misinterpret the phone's current position and orientation. (e.g., iOS screens enter a "call mode" when held up to the ear, or screen rotation will change what options are available. Even on new iOS and Android devices, sometimes the OS just seems to get "stuck" in a particular orientation and only after considerable fiddling with the phone will it rotate to the intended orientation)
Likely, designers made the call that they could just avoid the entire mess by operating the entire voice assistant in speaker mode instead of trying to compete with other UX/UI features.
Additionally, I would assume that the idea of simply talking aloud to a device replicates the Sci-Fi computer experience for computer interaction, most notably in Star Trek in which crew members simply summoned the computer with "Computer: [plain language command or query]". Even outside of Star Trek, this was a fairly common idea for interaction with futuristic computer systems, and for enough of the population, this method of interaction is part of the cultural memory.
Siri used to have "raise to talk" that would activate Siri automatically if you put the phone to your ear outside a phone call, but from some googling it was removed in an update.
If you activate Google's voice recognition from a bluetooth headset, it will respond to you through the headset -- but you often have to interact with the device anyway, to get it to do more than the most simple of things.
Actually next time just try talking to it to complete the task.
I use Google's voice recognition a ton when on my motorcycle with a Bluetooth helmet. I've gotten in the habit of just trying things and to my surprise many have worked.
Play/pause music, mute/unmute voice navigation, ask how long until my next turn, ask it to navigate to somewhere else, call/text/send a hangouts message to a contact by their nickname, play a specific band or album of music, etc...
Likely, designers made the call that they could just avoid the entire mess by operating the entire voice assistant in speaker mode instead of trying to compete with other UX/UI features.
Additionally, I would assume that the idea of simply talking aloud to a device replicates the Sci-Fi computer experience for computer interaction, most notably in Star Trek in which crew members simply summoned the computer with "Computer: [plain language command or query]". Even outside of Star Trek, this was a fairly common idea for interaction with futuristic computer systems, and for enough of the population, this method of interaction is part of the cultural memory.