|
|
|
|
|
|
by kazinator
4331 days ago
|
|
|
What is my mistaken conclusion? Okay, so all known tools have been exhausted, so now you're down to people and their talent for finding bugs. What people should you pay? How do you know you're getting your money's worth out of those people? What if there really is nothing left to find: are you prepared to believe six months' worth of status reports which say "found nothing?" My point wasn't that only tools should be used; I put that in as an aside (wrapped in glaring parentheses!). If I hadn't, someone would have pointed it out for me in a reply: "Hey you fool, of course you can track whether people are really bug hunting and being honest about their activity, if they are using tools whose results are reproducible." Of course tools only find things that they are designed to find. My point was not at all that tools should be used because they will find the next Heartbleed, but rather that you have some hope of tracking the progress of a security team that is applying tools. The topic of submission isn't about what is the best way to find security holes, but about spending money on it. My view is that spending money wisely requires some definition of a "return on investment" and tracking of concrete goals. This is hard to do with security research (once tools-based approaches have been exhausted). |
|
|
I can't think of anything closer to "throwing money at FOSS" than something like the internet bug bounty. Google/Facebook/etc collected a pile of money and put it up for a bug bounty for software used by most of us on the internet. https://hackerone.com/ibb click through to the projects and look at all the bugs that have been rewarded. https://hackerone.com/internet and https://hackerone.com/sandbox are the coolest.
My interpretation of your general conclusion is: without quantification spending money/effort on security is not useful. I disagree with that because its the nature of the beast. Its useful to have people look through code and some weeks there will not be a lot of findings. Its absolutely okay for a status report to read "I tried this, thought think might work, investigated the way X works to ensure it doesn't do Y - 0 total findings".
What people to pay & how to know you are getting your moneys worth are not unsolvable problems. For example at the company I work with we hold yearly bake-offs giving different security consultants the same code to see what bugs they find, we then use the best 2 or 3. Thats an approximation sure, but it solves your what people to pay problem.
How to know if you are getting your moneys worth, this is harder and rubs against the essence of security/QA work. No one knows what lurks in randomCode.tar.gz. That is the whole point of the exercise. But apparently the world agrees its useful to have corporate application security teams to do some vetting of the code looking for vulns, more useful that nothing at least. More useful than tools? Well thats a weird comparison because you likely need security people (or engineers with a bit of security background at least) to run some tools. I think tools vs people is a different debate but I would bet on people even at an equal cost point.
I agree quantification of security research is hard, I disagree that because we can't quantify something it is not useful.