[borski]

For what it's worth, PCI compliance as it stands today is complete BS. It provides a false sense of security and most of the PCI ASVs are the scourge of infosec. I can't tell you how many customers we have that use the cheapest possible PCI ASV for "compliance," but then use us in addition for "real security," despite the fact that we aren't an ASV. We've intentionally stayed away from becoming one thus far, actually, because that is a whole political game in itself. [1]

The new requirements are better. Stringent and hard to comply with, but better.

The real solution, as I see it, is to build automated security testing into your SDLC / Dev process. Penetration tests, when done by a good firm like Matasano, are incredibly useful, but lose their value the next time you push code. Building tools like Tinfoil into your CI process makes sure you don't get owned between pen tests.

PCI is unfortunately written by political minds and lawyers, not engineers or infosec folk. This is an unpopular comment, but is true in my estimation. Comply because you must, but please don't treat it as the end-all be-all. Care about your customers' data just a little bit more.

[1] http://bretthard.in/2013/01/the-pci-asv-process-sucks/

[akshatpradhan]

TinFoil, I remember I invited you to speak at the Boston Security Meetup several years ago!

>The real solution, as I see it, is to build automated security testing into your SDLC / Dev process. Penetration tests, when done by a good firm like Matasano, are incredibly useful, but lose their value the next time you push code. Building tools like Tinfoil into your CI process makes sure you don't get owned between pen tests.

This is false because you're suggesting that security testing of your SDLC, a subset of the compliance program is a more diligent solution than the entire compliance program. I explain myself in a comment below and I'd be glad to debate with you: https://news.ycombinator.com/item?id=9510369

[borski]

Ha, good to see you here.

I'm not suggesting that there is absolutely no value to any of PCI. The fact that it forces you to think about security at all is already of some value. However, I am saying that passing a PCI audit is incredibly easy as compared to thorough automated testing, and especially compared to a (good) manual penetration test. Because you can pass a PCI audit relatively easily, people will do that and think it's enough, when in reality there is far more they should be doing to protect their customers.

Should you not do PCI? No, of course you should, as it's required by the processors. Is PCI going to protect you from getting breached? No, it's not enough, and you shouldn't pretend it is.

If you have to pick exactly /one/ thing to do in addition to (or instead of) PCI, building thorough automated security testing into your SDLC process is it.

[akshatpradhan]

>If you have to pick exactly /one/ thing to do in addition to (or instead of) PCI, building thorough automated security testing into your SDLC process is it.

I don't understand how SDLC secure testing is an addition to PCI when its really a sub requirement of PCI (Req 6 which addresses SDLC and secure code testing)

I'm going to rephrase because I'm still confused: You're saying that in addition to doing PCI, I should do a sub requirement of PCI.

Why does that sound like circular logic to me?

[borski]

Show me where it suggests automated testing as a regular part of your SDLC. It recommends applying patches, coding to secure guidelines / best practices, doing code reviews, and running an automated or manual pen test at least once a year. Nowhere does it state in Requirement 6 that you should build automated testing into your SDLC.

Reference: https://www.pcisecuritystandards.org/documents/pci_dss_v2.pd...

[akshatpradhan]

Just so we're clear, what do you define as automated testing? Let me know.

[borski]

Scheduled testing that happens on an automatic basis, or that occurs whenever code is deployed or committed. For example, whenever you would run your unit tests or integration tests, you should also run security tests.

It's the difference between doing an automated or manual penetration test every 12 months and testing your application for vulnerabilities with every deploy.

[jerematasno]

> Penetration tests, when done by a good firm like Matasano, are incredibly useful, but lose their value the next time you push code.

I'd like to nicely but firmly push back on this one, and have longitudinal analysis of clients' applications to back it up. We put a lot of effort into helping our customers improve over time, both formally (writing helpful recommendations) and informally (educating developers during and after the test). There exist customers that ignore our advice, and don't improve, but most have a dramatic improvement in new code quality after the first assessment, and continue to year after year.

[borski]

Ah, you misunderstood what I meant. I didn't mean to imply that penetration tests, when done well, have no lasting value. I simply meant to imply that without a code freeze, there is always the chance of a new vulnerability creeping in no matter how well you follow checklists, best practices, or retain knowledge about types of vulnerabilities and how not to build them.

For that reason, automated testing on a continuous basis is important.

This is the same reason that you don't QA an application once a year. UIs change, requirements change, and for that you write integration tests, unit tests, etc.

Does that clarify things a bit? I didn't mean to imply Matasano did a poor job of educating their customers; in fact, I think you're among the best.

[vbezhenar]

The real security audit should be done by hackers the same way browser and OS vendors do it. Vendor lists his website on some platform and specifies money he's willing to pay for found vulnerability. Hackers trying to find vulnerabilities. 3-rd party verifies vulnerability and ensures that hackers are paid. More money vendor offers for vulnerability — more hackers trying to crack his site — more confidence clients have.