This would scale very badly, correct? If you have to hijack the connection for each client and then spend awhile doing a task (waiting for a task to complete, etc), then your whole webserver thread is going to be sitting there waiting.
That depends on the application server's concurrency model. If you have enough threads then it won't be a problem. While threads don't have the "web scale to 100,000 users!!!" reputation, you can still get very far with them on most workloads.
On evented application servers, you can integrate with the event loop API.
And, I'm not 100% sure whether the spec allows this, but it would appear that on application servers that are not evented, you can even offload the socket to a thread running an event loop. For example Phusion Passenger's implementation allows this.
I wonder if Heroku will add support for this. It'd be pretty nice to be able to just deal with arbitrary sockets there instead of being limited to just HTTP.
It depends on the webserver you use, not Heroku. The webserver must implement the hijacking API of Rack.
Also, you don't necessarily have to run an HTTP server on Heroku. They'll run whatever you want (as long as you change your procfile), so I think you can just go nuts. I think the only restriction is that you can't have the socket open for more than 30 seconds (though that may be 30 seconds of the socket being idle).