[sjtgraham]

> The worst offenders however are the requirements that some businesses simply cannot comply with unless they have some serious cash laying around. Examples of this are

>> Quarterly external vulnerability scans must be performed by an Approved Scanning Vendor (ASV), approved by the Payment Card Industry Security Standards Council (PCI SSC).

> and

>> Is external penetration testing performed per the defined methodology, at least annually, and after any significant infrastructure or application changes to the environment (such as an operating system upgrade, a sub-network added to the environment, or an added web server)?

Have you ever had a penetration test done? They basically run a load of OSS automated tools, generate a PDF report, and then charge you $1000s. It gives you no real insight and reveals nothing unless you've been a total noob. Why is this so expensive?

Broadening of PCI scope + needlessly expensive compliance = Smells like a large opportunity.

[dsacco]

I'm sorry that's been your experience. What should have happened is a primarily manual penetration test, administered by security engineers who themselves used to be fully competent software developers. Any automated tooling should have had strict manual verification and should not have been the focus of the test. Furthermore, superfluous results should not have been submitted in the PDF report.

Strictly "policy" audits such as PCI compliance differ a bit, but in general they should still involve a technical deep dive into your product's infrastructure, conducted by consultants with expertise in multiple tech stacks and overall experience in a variety of frontend and backend frameworks.

The final deliverable ("PDF report") also should have been hand-written, and in language that conveys technical expertise, complete with recommended steps towards remediation of any issues.

My employer, Accuvant does this, as well as Matasano (more well known here on HN).

As for why it's so expensive...well, I bill out at about $2000 per day. It really comes down to a lot of what people like patio11 and tptacek like to talk about here regarding consulting:

1. This is highly specialized work, with a much smaller population of competent engineers than typical web developers (for example). As such, it naturally receives a higher fee for supply and demand. Now, some people abuse this and run scans like Nessus and call it a day. These are not real infosec firms, they are parasites.

2. More specifically, we ask for it and we receive it, and we do exceedingly well. If people keep paying us five figures a week to perform a penetration test, we're not going to stop asking for it or reduce our prices.

[hueving]

>These are not real infosec firms, they are parasites.

The entire consulting penetration testing market is setup to encourage this behavior. There is no way to prove you actually did anything correct. Someone can write a wonderful PDF analysis by hand and still leave the system full of glaring holes. Customers can't tell a system is broken until it gets hacked.

>More specifically, we ask for it and we receive it, and we do exceedingly well. If people keep paying us five figures a week to perform a penetration test, we're not going to stop asking for it or reduce our prices.

Right, but many times I've seen companies do it because they are desperate to do it for compliance purposes. :/ Essentially there is a non-trivial portion of the market held up by regulatory demand.

>Strictly "policy" audits such as PCI compliance differ a bit, but in general they should still involve a technical deep dive into your product's infrastructure, conducted by consultants with expertise in multiple tech stacks and overall experience in a variety of frontend and backend frameworks.

I'm curious. Do you review every line of code in a customer's codebase? What about the code of every library they import? If you don't review imports, do you leave a big caveat in your report that says their code looks okay, but the libraries could be full of vulnerabilities?

[raesene4]

Whilst I'm not the parent commenter I do work in the same industry..

>>These are not real infosec firms, they are parasites.

>The entire consulting penetration testing market is setup to encourage this behavior. There is no way to prove you actually did anything correct. Someone can write a wonderful PDF analysis by hand and still leave the system full of glaring holes. Customers can't tell a system is broken until it gets hacked.

So a good company should be able to provide a methodology detailing the tests they do, you'll also see some who report the tests conducts and the results (positive or negative), so asking for sample reports from consultancies would help to find one closer to your specific needs. Personally I prefer reporting all test results as it keeps both parties straight on what has and has not been covered.

>>More specifically, we ask for it and we receive it, and we do exceedingly well. If people keep paying us five figures a week to perform a penetration test, we're not going to stop asking for it or reduce our prices.

>Right, but many times I've seen companies do it because they are desperate to do it for compliance purposes. :/ Essentially there is a non-trivial portion of the market held up by regulatory demand.

Yeah where people are getting tests for purely compliance reasons there is a tendency to go with cheap suppliers as there's not really good perceived benefit.

>>Strictly "policy" audits such as PCI compliance differ a bit, but in general they should still involve a technical deep dive into your product's infrastructure, conducted by consultants with expertise in multiple tech stacks and overall experience in a variety of frontend and backend frameworks.

>I'm curious. Do you review every line of code in a customer's codebase? What about the code of every library they import? If you don't review imports, do you leave a big caveat in your report that says their code looks okay, but the libraries could be full of vulnerabilities?

Heh this is one of the huge gaping holes in security at the moment. Most applications are now constructed of piles of code acquired from repos (npm, nuget, rubygems etc) that provide absolutely no curation of content and anyone can put any code they like up there. There is (from what I've seen) very little appetite from companies to actually try and audit their whole stack, generally due to the cost of doing that. Manual code review is expensive and when you start importing 100Kloc of 3rd party code into your solution it would not be a cheap excercise to validate...

[nsxwolf]

Can you elaborate on what manual tests are and why they are better? Coming from a software QA perspective, automated tests provide repeatability and prevent omissions and aid with regression testing.

What are "manual tests" in the infosec context?

[metaverse]

We went through a SAQ D Service Provider 3.0, and paying for an ASV didn't hurt nearly as much as filling out that 80 page questionnaire... In fact, it reminded us apply some recent CVE's to our system before taking it to production. We used Comodo HackerGuardian which is $250/y, so you don't have to pay $1000s.

[akshatpradhan]

I think my tool can actually help you. You document all your procedures in an organized structure using Github flavored markdown. http://cc-stg2.herokuapp.com/compliancechimp/documents/softw...

We're becoming the true Turbo Tax for compliance.

[tomjen3]

You probably have a good tool, but I am not even going to take a look at it when it doesn't have its own url. herokuapp just screams weekend project.

[akshatpradhan]

I do have a real tool. Its on ComplianceChimp.com. But its down right now because my TLS Certs expired a couple days ago and I never recorded my Key rotation procedures. I did think about temporarily disabling SSL, specifically for this HN post, but thought against it. We want to do compliance right.

We're fixing the certs now and updating our Key rotation procedures for you all to see in our publicly viewable compliance workbook.

[ceequof]

Not to kick someone while they're down... but a security company screwing up their key rotation is not exactly a good sign.

[akshatpradhan]

There's only so much I can get done with my dwindling runway =(.

But you're right, that getting Key Rotation Procedures documented is the 1st thing I should have done.

This is good feedback actually because now I know that after scoping the assets in my turbo tax-like tool, the very next thing a person should do is write down their key rotation procedures. Its also easier to write out as a procedure because its such a common yet forgetful practice.

We're putting cycles into this right absolutely now.