[skytrue]

Sorry, I was non-specific. If you're using ChatGPT, you're basically using a "product" that OpenAI created. It has specific system prompts and prompt engineering to ensure that it stays on the rails. If you, instead, use the OpenAI API's for GPT-3.5/GPT-4, you aren't beholden to the ChatGPT "product". It's very easy to create a chatbot (not using ChatGPT, the product) that only produces JSON. It's just hard to get ChatGPT to do that same activity.

That's why everybody doing experiments in this space should either 1) be using the OpenAI Playground, or 2) using the API, and not using ChatGPT.

[verdverm]

Ah ok, I'm primarily using the API already. One interesting thing is that the GPT-3.5 "product" is much faster, but looks to be using a different model in the request, their encoding model iirc. I wonder if they are now using embeddings to cache results to reduce load on the real models when they can?