In theory, you could fingerprint the host OS first and then run the appropriate commands (of course more tricky with more custom Linux setups, does CTRL+ALT+Fn still work to get to a text console?): https://www.cise.ufl.edu/~butler/pubs/sadfe11.pdf
Yeah, I was thinking of custom window-manager setups. You can usually get a tty console by ctrl-meta-f1 etc., but that wouldn't help, since you'd have to enter a password. I suppose an advanced version could try different combinations and test each by entering a command that would be detected by the stick.