Hacker News new | ask | show | jobs
by tom1337 650 days ago
OpenAI recently announced a Batch API [1] which allows you to prepare all prompts and then run them as a batch. This reduces costs as its just 50% the price. Used it a lot with GPT-4o mini in the past and was able to prompt 3000 Items in less than 5min. Could be great for non-realtime applications.

[1] https://platform.openai.com/docs/guides/batch
3 comments
Tostino 650 days ago
I hope some of the opensource inference servers start supporting that endpoint soon. I know vLLM has added some "offline batch mode" support with the same format, they just haven't gotten around to implementing it on the OpenAI endpoint yet.
link
asaddhamani 650 days ago
Do note it can take up to 24 hours or drop requests altogether. But if that’s not an issue for your use case it’s a great cost saving.
link
jumploops 650 days ago
This is neat, I’ve been looking for a way to run our analytics (LLM-based) without affecting the rate limits of our prod app.

May need to give this a try!

link
altdataseller 650 days ago
What percentage of requests usually get dropped? Is it something miniscule like 1% or are we talking non trivial like 10%
link
johndough 650 days ago
llama.cpp enabled continuous batching by default half a year ago: https://github.com/ggerganov/llama.cpp/pull/6231

There is no need for a new API endpoint. Just send multiple requests at once.

link
Tostino 650 days ago
The point of the endpoint is to be able to standardize my codebase and have an agnostic LLM provider that works the same.

Continuous batching is helpful for this type of thing, but it really isn't everything you need. You'd ideally maintain a low priority queue for the batch endpoint and a high priority queue for your real-time chat/completions endpoint.

Would allow utilizing your hardware much better.

link
LunaSea 650 days ago
That's a great proposition by OpenAI. I think however that it is still one to two orders of magnitude too expensive compared to traditional text extraction with very similar precision and recall levels.
link
cdrini 650 days ago
Yeah this was a phenomenal decision on their part. I wish some of the other cloud tools like azure would offer the same thing, it just makes so much sense!
link