[jfisher4024]

Congratulations on the launch! I was unable to use this library: I was trying to evaluate different non-OpenAI models and it consistently failed due to malformed JSONs coming from the model.

Any thoughts about using different models? Is this just a langchain limitation?

[shahules]

Thanks for your feedback. We have tested Ragas on alternatives like Claude, Mixtral, Gemini, etc.

Although we support all LLMs supported by Langchain, sadly many of the OSS models out of the box aren't capable of generating JSON-type output which is important for us to ensure reproducibility.

[jfisher4024]

Any tips for Mixtral? That’s what we tried

[shahules]

Hey, I would recommend checking out our PRs. There would be PRs that have modified some of the prompts to better suit Mixtral.

[3d27]

Checkout this instead: https://github.com/confident-ai/deepeval

Also has native ragas implementation but supports all models.