|
|
|
|
|
|
by rawsh
653 days ago
|
|
|
Bit confused what the value add is over a framework like DSPy. This still requires you to create an eval dataset with ground truth, basically the only hard part of using DSPy. Easily getting the optimized prompt and having some metrics out of the box is not worth nearly $1k/mo IMO Side note: I’ve had a lot of luck combining automatic prompt optimization with finetuning. There is definitely some synergy https://raw.sh/posts/chess_puzzles |
|
|
1. You are absolutely right, the dataset is a big hurdle for using DSPy. That's why we offer a synthetic dataset generation pipeline for RAG, agents, and a variety of LLM pipelines. More here: https://docs.relari.ai/getting-started/datasets/synthetic
2. Relari is an end-to-end evaluation and optimization toolkit. Real-time optimization is just one part of our data-driven package for building robust and reliable LLM applications.
3. Our tools are framework agnostic. If you can build your entire application on DSPy, that's great! But often we see AI developers hoping to maintain the flexibility and transparency to have their prompts / LLM modules work with different environments.
4. We provide well-designed metrics and/or custom metrics learned from user feedback. We find good metrics very key to making any optimization process (including prompts and fine-tuning) work.