[neelm]

Something like this is going to be needed to evaluate models effectively. Evaluation should be integrated into automated pipelines/workflows that can scale across models and datasets.

[krawfy]

Thanks Neel! We totally agree that automated evals will become an essential part of production LLM systems.