Hacker News new | ask | show | jobs
by gberger 1000 days ago
That looks like a timeout configured on the web servebr
2 comments

Yes, it seems their servers are terminating responses early. Unfortunately it's undocumented, was introduced unexpectedly one day, and prevents full use of larger context-window models.

It also charges in full for the entire response, which seems to continue running in the background. Definitely a bug at their end.

Since it's 5 minutes I'm wondering if it's just "that's how the web works by default". I remember a while ago I was doing a long-running calculation on an POST request and was trying to debug why the connection always broke after 5 minutes, and found the hard way that there's a 5 minute timeout.
Seems like a means to avoid DoS?
I think it's just some networking misconfiguration they haven't noticed. I can't see them intentionally trying to prevent DoS this way, that's what API key rate limits are for (it would be a very expensive and slow DoS otherwise!).
>that's what API key rate limits are for

You don't need an API key to make a connection to the API.

Very likely. Typical serverless hosts like Vercel times out at 5 minutes.