[gcr]

i basically can't use the ChatGPT app on the subway for these reasons. the moment the websocket connection drops, i have to edit my last message and resubmit it unchanged.

it's like the client, not the server, is responsible for writing to my conversation history or something

[spruce_tips]

it took me a lot of tinkering to get this feeling seamless in my own apps that use the api under the hood. i ended up buffering every token into a redis stream (with a final db save at the end of streaming) and building a mechanism to let clients reconnect to the stream on demand. no websocket necessary.

works great for kicking off a request and closing tab or navigating away to another page in my app to do something.

i dont understand why model providers dont build this resilient token streaming into all of their APIs. would be a great feature

[rishabhaiover]

exactly. they need to bring in spotify level of caching of streaming music that it just works if you're in a subway. Constant availability should be table stakes for them.

[rjzzleep]

I get that the web versions are free, but if you can afford API access, I always recommend using Msty for everything. It's a much better experience.

https://msty.ai/