[sebastiennight]

Interestingly enough, I don't think this applies to the APIs as much.

What I've seen on indie hacker type website is that developers are fully on this train and not very critical of the outputs.

This is why you get very basic prompts sent by "wrapper apps", which might have given the developer a good result the only time it was tested before being put in production.

I think it might take a while before tools show up that can generate 100 test cases and test a given prompt with all 100 to report on the results... It seems to be a tough problem to crack.

IMHO front-end chat end-users have many many more "at-bats" and get to see more model results than devs do, which make them more critical of those results.