[advaith08]

imo they dont have batching because they pack sequences before passing through the model. so a single sequence in a batch on OpenAI might have requests from multiple customers in it

[sidnb13]

Ah that would make sense. Similar to vLLM which does dynamic packing.