[WalterGR]

There is no universal metric for security.

What about slashdot-style raw counts of vulnerabilities? As Microsoft got their security more together, it seems like that measure has fallen out of favor.

I am frequently astounded by how many vulnerabilities Firefox point releases address, but I don't know the stats.

[geofft]

That gives actively misleading information in two cases: if two vendors have different standards for vulnerabilities (e.g., is a tab crash a DoS or just a thing that happens sometimes) and if one vendor is more active at actually looking to find vulnerabilities (or incentivizing others to do so) than another. Both of those problems, inverted, make the metric easily gameable in awful ways: you're encouraging vendors to have a high bar for what counts as a security bug and you're encouraging them not to look.

A little more fundamentally, vulnerability counts don't provide a great measure of security architecture: whether defense-in-depth and least privilege are good at mitigating the impact of bugs. If, say, you have a JS PNG decoder and once a year someone finds a way for a malicious PNG to run arbitrary JS within the usual web JS sandbox, that's probably way better than having an unsandboxed native-code PNG decoder and once every three years someone finds a way for a malicious PNG to run arbitrary code on the host. (Maybe you can do something involving weighting vulnerabilities by CVSS score or something.)

[quotheth]

Vulnerability counting is never a metric for security when comparing counts across products.

It's very helpful for counting within a product.

One simple example - A and B are browsers. A has a bounty program that they invest millions into, B does not.

A gets 100 vulnerability reports a month, B gets 5.

Is B safer?

No.

However, let's say we just look at A. It gets 100 a month, and then a new feature is added - suddenly it's 200 a month, and the vulns tend to be in the new codebase. That's interesting information for A - it has nothing to do with B.