Hey all, my friend and I were building a web app that involves calling an LLM and ran into the question of testing. Although there are many great solutions out there, we couldn't find one that checked all of these boxes:
- Not configuration driven, ex. YAMLs
- Native to the JS / TS testing ecosystem
- Developer oriented, so not including unnecessary UI components
- Simple, familiar API
- Makes it easy to write use case specific tests
- Free and open source (MIT License)
This is why we created Poyro (https://docs.poyro.dev/). Poyro brings the world of LLM feature evaluation into that of full-stack unit testing. Like you would with any other unit tests, you can run them for free locally on your machine with Poyro. Iteration velocity is critical to any type of software engineering, and LLM features should be no different. Poyro removes the friction of submitting test cases to costly, remote models. It also removes the friction of treating your unit tests for LLM features any differently from your other unit tests, now they can live together side by side.
Poyro is a very light and simple extension to Vitest, which is a modern Jest-like framework for unit testing. It uses llama.cpp under the hood to run a quantized version of Llama-3 so you can define natural language conditions to check your LLM outputs against. We have intentionally kept the API extremely simple (one single method) to make it really easy to use. We'd love to hear your feedback to learn the best ways to extend the API or otherwise improve the library. If you want feedback or have any questions you can reach us at our Discord (https://discord.gg/gmCjjJ5jSf).
- Not configuration driven, ex. YAMLs
- Native to the JS / TS testing ecosystem
- Developer oriented, so not including unnecessary UI components
- Simple, familiar API
- Makes it easy to write use case specific tests
- Free and open source (MIT License)
This is why we created Poyro (https://docs.poyro.dev/). Poyro brings the world of LLM feature evaluation into that of full-stack unit testing. Like you would with any other unit tests, you can run them for free locally on your machine with Poyro. Iteration velocity is critical to any type of software engineering, and LLM features should be no different. Poyro removes the friction of submitting test cases to costly, remote models. It also removes the friction of treating your unit tests for LLM features any differently from your other unit tests, now they can live together side by side.
Poyro is a very light and simple extension to Vitest, which is a modern Jest-like framework for unit testing. It uses llama.cpp under the hood to run a quantized version of Llama-3 so you can define natural language conditions to check your LLM outputs against. We have intentionally kept the API extremely simple (one single method) to make it really easy to use. We'd love to hear your feedback to learn the best ways to extend the API or otherwise improve the library. If you want feedback or have any questions you can reach us at our Discord (https://discord.gg/gmCjjJ5jSf).
If you enjoy using Poyro please give us a star on Github! (https://github.com/poyro/poyro)