In practice, things like Base64 encoding are used. That's what we use in Ganeti Web Manager to host a noVNC (HTML5 VNC client) and have it talk WebSockets with the backend.
In theory, there's no good reason we can't have binary frames; I have internally filed it under the "Why can't we have nice things?" category.
https://labs.ericsson.com/apis/web-real-time-communication
http://news.ycombinator.com/item?id=2600585