Hacker News new | ask | show | jobs
by zdragnar 201 days ago
As a counter anecdote, I've yet to try a model where I break even. The output is so untrustworthy that I need to spend as much time coaching and fixing as if I'd just written it myself. When it does produce a working result faster, I still end up with less confidence in it.

What I'm working on doesn't have much boilerplate, though, and I've only been programming for 18 years. Maybe I need to work for another 7 before it starts working for me.
2 comments
AYBABTME 200 days ago
There's definitely a method to using them well. It took me 6 months of trial, giving up, trying again, refining my approach, ... to eventually get fairly good results in a consistent manner. It's useful to know what the LLMs are good at and what type of errors they will do. It's also very useful to be a stickler about software engineering practices to keep the LLMs focused in the right direction.

Example stuff that helps:

  - extensive test suites
  - making LLMs use YAML for data-intensive files, instead of writing inline
  - putting a lot of structural guard rails using type-systems, parse-dont-verify, ...
  - having well scoped tasks
  - giving the LLM tight self-serve feedback loops
Recently I made it fix many bugs in a PEG grammar and it worked really well at that. I made it turn a test suite from an extensive Go test array to a "golden file" approach. I made it create a search index for documentation and score the search quality using qualitative IR metrics, and then iterate until the metrics met a minimum standard.
link
mring33621 201 days ago
Its effectiveness is going to depend on the domain and tech stack used.

You also need to know what chunks of AI output to keep and what to throw away.

For example, Gemini 'Fast' quickly identified a problem for me the other day, based on the stacktrace. But its proposed solution was crappy. So, I was happy with the quick diagnosis, but I fixed the code manually.

link