Anything that Comet or other long-polling techniques provide, really.
The advantages of WebSockets include constantly changing specifications which are neither backwards- nor forwards-compatible, a lack of proxy traversal, an arbitrary and unreasonable limitation on binary data transfer, and lack of unified browser and server support.
It's not that bad. Right now you basically have support for the -76 protocol in release channel Chrome and Firefox 4 or 5, it's nearly trivial to support all of those with a single server.
With Firefox 6.0, I had to spend some time to upgrade our little Python server, but now it happily serves both -76 and -07. It also looks like -07 is very close to being signed off on as the actual standard, so the compatibility problems are probably a thing of the past soon (-76 and -07 are actually completely trivial to discern from a server point of view).
I haven't tried with binary data, but it looks like it's explicitly supported with -07. Not sure about proxies, they aren't relevant to our work app.
For server side development, I've found that Jetty has a good websockets API. I had difficulty getting the server and browser to agree a few months ago when I tried it (common libs in python and C# were tested). Moving to Jetty got it working. I'm not sure how current it is though.
In practice, things like Base64 encoding are used. That's what we use in Ganeti Web Manager to host a noVNC (HTML5 VNC client) and have it talk WebSockets with the backend.
In theory, there's no good reason we can't have binary frames; I have internally filed it under the "Why can't we have nice things?" category.
WebSockets will generally provide latency improvements over long-polling as they won't have the overhead of creating and breaking down many HTTP connections (as to "push" using long-polling you need to return to the browser). XHR-multipart communication is better than long-polling, but only Firefox supports it.