Hacker News new | ask | show | jobs
by alganet 205 days ago
> "Take a look at those tests!"

A math module that is not tested for division by zero. Classical LLM development.

The suite is mostly happy paths, which is consistent with what I've seen LLMs do.

Once you setup coverage, and tell it "there's a hidden branch that the report isn't able to display on line 95 that we need to cover", things get less fun.
1 comments
mjaniczek 205 days ago
It's entirely happy paths right now; it would be best to allow the test runner to also test for failures (check expected stderr and return code), then we could write those missing tests.

I think you can find a test somewhere in there with a commented code saying "FAWK can't do this yet, but yadda yadda yadda".

link
alganet 205 days ago
It's funny because I'm evaluating LLMs for just this specific case (covering tests) right now, and it does that a lot.

I say "we need 100% coverage on that critical file". It runs for a while, tries to cover it, fails, then stops and say "Success! We covered 60% of the file (the rest is too hard). I added a comment.". 60% was the previous coverage before the LLM ran.

link