Hacker News new | ask | show | jobs
by lelandfe 70 days ago
Yes, and with very long chats, you'll see it even forget how to do things like make tool calls - or even respond at all! I've had ChatGPT reply with raw JSON, regurgitate an earlier prompt, reply with a single newline, regurgitate information from a completely different chat, reply in a foreign language, and more.

Things get really wacky as it approaches decoherence.
1 comments
kayodelycaon 70 days ago
I’ve seen the raw json before. I didn’t realize that was an actual failure mode.

I’ve also had it fail to respond in long chats but I thought it was a network error despite having no error messages.

link
lelandfe 70 days ago
Yeah, the raw JSON (in my case) is the result of a failed tool call, it was trying to generate an image. With thinking models, you can observe the degeneration of its understanding of image tool calls over the lifetime of a chat. It eventually puzzles over where images are supposed to be emitted, how it's supposed to write text, if it's allowed to provide commentary - and eventually, it gets all of it wrong. This also happens with file cites (in projects) and web search calls.
link