[araghuvanshi]

The cost comparisons for the same model are interesting. I'm curious about why certain providers are a lot cheaper than others - for example, mistral-8-7b on OctoAI costs $0.2/1m tokens whereas it's $0.66 using mistral's own inference. My best guess is that one includes a cold start. Any thoughts there?

[danlenton]

I think different providers are just trying to provide value at different areas in the market. OctoAI are consistently the most cost effective, but typically not as fast, while others are fast but come at a premium. In general, some providers are also willing to operate with little or no positivie margins to gain traction. I think things might stabilize more over time. Will be interesting to see!