Hacker News new | ask | show | jobs
by gchamonlive 12 days ago
Not my experience. It does depend on the scale of the project, but for tens of thousands LOC, it's really good at taking instructions for architectural redesign.

LLM will take stuff at face value. If you tell it it's good, it'll gladly abide. Similarly if say it's crap it'll so the complete opposite.
1 comments
whstl 10 days ago
Yes, it's good at taking instructions for architectural redesign, but when it's not doing an actual architectural redesign task, it will be heavily biased towards the status-quo, despite AGENTS.md and prompts trying to steer it.

My latest experience was towards a C++ to Rust migration and it failed miserably because the existing codebase enforced patterns I didn't want.

link
gchamonlive 10 days ago
Ah yes, I understand, but that's expected and actually desirable, it's preferable than having inconsistent project patterns across the project introduced by multiple LLMs. Imagine the mayhem.

It's important to do a first pass cleaning up and preparing code before unleashing agents onto it, similar to how Working Effectively with Legacy Code advocates approaching consolidated codebases. After that the agent will actually pick up the new pattern and start propagating that to the rest of the code.

link
whstl 10 days ago
I believe it’s desirable as a default, but not when the prompting/steering asks not to.

> It's important to do a first pass cleaning up and preparing code before unleashing agents onto it

That’s a good idea in general, but the whole point of the rewrite was that Fable wasn’t capable enough of fixing the old codebase which we were using as a reference.

Another problem I had in another project was with having “sample” code purely written for testing being used as the architectural guidance, despite comments and repeated memories asking not to.

In the end, by the way, the solution was to simply blocking access to specific files, in the Claude settings.json.

But still Claude tried to cheat by running ‘cat file.cpp` a few times, so…

link
gchamonlive 10 days ago
That's really interesting. I'll be more careful with RAG in the future then, thanks for the account
link