Hacker News new | ask | show | jobs
by jw1224 1000 days ago
Yes, it seems their servers are terminating responses early. Unfortunately it's undocumented, was introduced unexpectedly one day, and prevents full use of larger context-window models.

It also charges in full for the entire response, which seems to continue running in the background. Definitely a bug at their end.
2 comments
yossi_peti 1000 days ago
Since it's 5 minutes I'm wondering if it's just "that's how the web works by default". I remember a while ago I was doing a long-running calculation on an POST request and was trying to debug why the connection always broke after 5 minutes, and found the hard way that there's a 5 minute timeout.
link
pbhjpbhj 1000 days ago
Seems like a means to avoid DoS?
link
jw1224 1000 days ago
I think it's just some networking misconfiguration they haven't noticed. I can't see them intentionally trying to prevent DoS this way, that's what API key rate limits are for (it would be a very expensive and slow DoS otherwise!).
link
charcircuit 1000 days ago
>that's what API key rate limits are for

You don't need an API key to make a connection to the API.

link