[spencerchubb]

I also could not find any mention of the methodology.

[sitkack]

Without this detail, the whole body of work is anecdotal.

[benreesman]

To be fair, most of the commentary on both sides of the LLM conversation are pretty anecdotal, which is increasingly looking like a structural problem given that any solid evidence goes in the training set in about an hour.

[gctwnl]

Definitely. Otherwise it would have required a lot more than a single blog post. It is an observation, not anything rigorous with a large number of examples, and decent statistics.

[sitkack]

It doesn’t require a whole more. 1) the full transcript of the llm exchange and 2) version numbers for the llm would go a long way.

It is basically a long winded way of saying in a bug report, “it doesn’t work”.