[swyx]

congrats on launching! i think my continuing struggle with looking at Ragas as a company/library rather than a very successful mental model is that the core of it is like 8 metrics (https://github.com/explodinggradients/ragas/tree/main/src/ra...) that are each 1-200 LOC. i can inline that easily in my app and retain full control, or model that in langchain or haystack or whatever.

why is Ragas a library and a company, rather than an overall "standard" or philosophy (eg like Heroku's 12 Factor Apps) that could maybe be more universally adopted without using the library?

(just giving an opp to pitch some underappreciated benefits of using this library)

[shahules]

Thank you for asking this question.

To answer this question, I will explain two directions of Ragas.

The first one is the horizontal expansion of the library which involves features like

- Giving you the ability to use any LLMs instantly without any hassle

- Asynchronous evaluations, integrations with tracing tools, etc

- Automatic support to adapt metrics to any language

The second is vertical expansion or adding more core features like metrics to Ragas which includes.

- Synthetic test data generation: this is something that is heavily loved by our community so we are continuously improving the quality of it. https://docs.ragas.io/en/stable/concepts/testset_generation....

Now, as we expand in both directions we aim to solve the problem of how to evaluate and test compound systems. Now, to solve this we will be innovating and working on features like feedback utilization, automatically synthesizing assertions, etc to solve this hard problem.

I hope I was able to answer your question. Would love to discuss more.

[swyx]

cool cool. so 1) will be a direct langchain competitor, and 2) is net new territory?

[shahules]

1) Horizondal expansion and support are core to every framework/library. This won't make us a competitor to LC, we actually use langchain-core to support many of these like supporting different LLMs. 2) We operate in a layer underneath the stack of evals and testing because we want to solve the problem from the ground up rather than building a fancy tracing tool, which comes later in the stack.

[rolisz]

> that are each 1-200 LOC. i can inline that easily in my app and retain full control

Isn't that true of most of langchain as well though?

[shahules]

I think it's true for any early-stage library/framework. The tradeoff is then you will have to keep maintaining it, add support to other LLMs if you change LLMs, etc. Then in the end OSS will be far ahead because by that time it will have smoothened its rough edges.

[rolisz]

Or OSS will be going in a different direction then what you need, so if you are using it you'll either be stuck on an old version or you have to keep fighting around it. ML libraries in particular have this annoying habit of not being very backwards compatible over more than 2-3 years.