[shadowfacts]

When people write blog posts about how LLMs failed for some particular task, the responses from boosters invariably fall along the lines of "just use this other model/just tweak your prompt like so/you're just not skilled enough—you can't make fundamental arguments about AI by citing specific examples."

So we can't make arguments by citing specific examples, and also can't make arguments by not citing specific examples. Whelp, I guess that's the ball game.

(yes yes, I'm committing a group attribution error, but still)

[gymbeaux]

I think we should investigate the backgrounds of those making claims one way or another and rely on those backgrounds for determining credibility. I suspect that we'd find that those who are saying LLMs write great, bulletproof code with "100% unit test coverage" (true story- a coworker was bragging about 100% unit test coverage) are not really qualified to be software engineers. This is a trend I have noticed in my org. Those drinking the most LLM kool aid do NOT have an engineering/comp sci degree, have relatively little experience, resumes are incredibly weak (e.g., generic stuff that we've all done as software engineers).

We no longer have the luxury of welcoming bootcamp engineers into our field with open arms. We need to protect our craft. Call these fools out or they'll keep spreading hype/FOMO.