Hacker News new | ask | show | jobs
by rl3 294 days ago
May I ask the rationale for writing your own? Were you using an existing tool that didn't quite fit your needs?

This is an itch I've been wanting to scratch myself, but the space has so many entrants that it's hard to justify the time investment.
1 comments
CuriouslyC 294 days ago
Existing off the shelf IR tools are mid, more recent research is often not productionized, and there are a lot of assumptions that hold for agentic context (at least in the coding realm, which is the one that matters) that you can take advantage of to push performance.

That plus babysitting Claude Code's context is annoying as hell.

link
rl3 294 days ago
Thanks.

>That plus babysitting Claude Code's context is annoying as hell.

It's crazy to me that—last I checked—its context strategy was basically tool use of ls and cat. Despite the breathtaking amount of engineering resources major AI companies have, they're eschewing dense RAG setups for dirt simple tool calls.

To their credit it was good enough to fuel Claude Code's spectacular success, and is fine for most use cases, but it really sucks not having proper RAG when you need it.

On the bright side, now that MCP has taken off I imagine one can just provide their preferred RAG setup as a tool call.

link
CuriouslyC 294 days ago
You can, but my tool actually handles the raw chat context. So you can have millions of tokens in context, and actual message that gets produced for the LLM is an optimized distillate, re-ordered to take into account LLM memory patterns. RAG tools are mostly optimized for QA anyhow, which has dubious carryover to coding tasks.
link
olejorgenb 293 days ago
> ... re-ordered to take into account LLM memory patterns.

If I understand you correctly, doesn't this break prefix KV caching?

link
CuriouslyC 293 days ago
It is done at immediately before the LLM call, transforming the message history for the API call.

This does reduce the context cache hit rate a bit, but I'm cache aware so I try to avoid repacking the early parts if I can help it. The tradeoff is 100% worth it though.

link
psadri 292 days ago
I’m curious about this project (I’m working on something similar). Anyway to get in contact with you?
link