[gruez]

That's not really a fair test because you're leading the model pretty hard, even if the prompt doesn't specifically say there's a bug to be found. It's basically the same objections that people raised in the thread where someone claimed current models are just as good as mythos.

[naruhodo]

I don't agree, and I'd like to understand your point of view.

To me, asking if a function has something wrong with it is just a very basic code review - something that should happen with every function. A competent, security conscious engineer would respond the same way as the model, unsurprisingly, since the model is... modelling competence.

[saagarjha]

Code review that finds problems in all code is useless.

[shay_ker]

right exactly, but clearly it's possible to elicit the behavior we want in the model, which means the capabilities are there!

[Matumio]

The more interesting question is, how many issues will this prompt report to you in random code that is perfectly fine?