|
|
|
|
|
|
by sig_kill
19 days ago
|
|
|
Yeah, this is one area we’re struggling with due to the sheer volume of variations and conditions, but I’ve been thinking of collecting some real-time statistics around latency, prefill/decode, and model distribution… that way we can update some kind of live + aggregated performance numbers for interested parties. |
|
|