Love the rigor of running an experiment many times to see how often it got the desired outcome. Most people would just fire off a prompt and be disappointed if the model didn’t find the bug!
Seems a clever technique for anything that needs strong defense against hallucination. Kind of an “average across runs”. Manually auditing results isn’t very scalable (cf. the author says they missed that the LLM caught the second half of the bug in some runs but they missed that detail). In future an LLM could do that bit too so the technique becomes scalable. One can imagine being given a meta-report of what’s in all the reports produced by the runs.
Based on the linked article I think Gerard would balk at this post, consider the content the exact kind of contribution that people would hate to deal with , and the headline “ceo weasel wording”.
Seems a clever technique for anything that needs strong defense against hallucination. Kind of an “average across runs”. Manually auditing results isn’t very scalable (cf. the author says they missed that the LLM caught the second half of the bug in some runs but they missed that detail). In future an LLM could do that bit too so the technique becomes scalable. One can imagine being given a meta-report of what’s in all the reports produced by the runs.