[danpalmer]

Several people have already raised the very good point that ultimately, we need the source code to be certain.

However, can we really be sure when we have the source? I don't think so. The codebase is likely to be large, especially when you start looking at dependencies such as the crypto libraries they may be using (unless you want to assume they are safe themselves), and it has been shown that humans are actually quite bad at finding vulnerabilities in code that is written to obscure its real purpose.

The Underhanded C Contest is a yearly contest that puts this to the test. Participants are given a spec for a small piece of software, and must write a program in C that appears on code review to work correctly, but in fact subverts the requirements in some way. This has been remarkably successful.

Sure, having the code is better than not having the code, but I think that gives us less security than many assume it does.

[ikeboy]

Would someone actually looking through those entries trying to find a problem fail? Or is it just "first glance doesn't show any problems" stuff? I thought it was 2.

[dllthomas]

It would be great to throw entries at actual security auditors, mixed with innocent versions, and see how they fare.

[ikeboy]

We need a name for this. How about Bug-complete Turing Test? Or just Buggy Turing Test.

[dllthomas]

I'm down for naming it, but I think "bug" is the wrong term for what we're talking about here, since we're talking about deliberate misbehavior.

[dllthomas]

"we need the source code to be certain."

The source and a lot of effort, or just a tremendous amount of effort.