[freediver]

Based on PyLLMs benchmark. [1]

Slightly better than Haiku and slightly slower. Much cheaper.

OpenAIProvider('gpt-4o-mini') Total Cost: 0.00385 | Aggregated speed: 105.72 tok/sec | Accuracy: 51.85%

AnthropicProvider('claude-3-haiku-20240307') Total Cost: 0.00735 | Aggregated speed: 117.53 tok/sec | Accuracy: 48.15%

[1] https://github.com/kagisearch/pyllms

[sauwan]

How long before Anthropic releases Claude-3.5-Haiku at the same price with significantly better performance? OpenAI in trouble...