[jerematasno]

> Penetration tests, when done by a good firm like Matasano, are incredibly useful, but lose their value the next time you push code.

I'd like to nicely but firmly push back on this one, and have longitudinal analysis of clients' applications to back it up. We put a lot of effort into helping our customers improve over time, both formally (writing helpful recommendations) and informally (educating developers during and after the test). There exist customers that ignore our advice, and don't improve, but most have a dramatic improvement in new code quality after the first assessment, and continue to year after year.

[borski]

Ah, you misunderstood what I meant. I didn't mean to imply that penetration tests, when done well, have no lasting value. I simply meant to imply that without a code freeze, there is always the chance of a new vulnerability creeping in no matter how well you follow checklists, best practices, or retain knowledge about types of vulnerabilities and how not to build them.

For that reason, automated testing on a continuous basis is important.

This is the same reason that you don't QA an application once a year. UIs change, requirements change, and for that you write integration tests, unit tests, etc.

Does that clarify things a bit? I didn't mean to imply Matasano did a poor job of educating their customers; in fact, I think you're among the best.