[numeri]

To be fair, it is good to know that it disobeys simple instructions like "don't examine my git history" far more than other models. (It should of course be a different benchmark, so as not to conflate things.)

It's not a great sign for alignment.

[bensyverson]

Agreed, alignment is just a separate issue that a vuln fixing benchmark doesn't need to be testing.