[sneak]

More troubling to me than the closed source firmware is that the bug in TFA seems like something that the most basic of a test suite should be catching.

It’s reminiscent of Apple’s “goto fail” lack of certificate checking - another easily testable case that simply wasn’t.

The test authors don’t even need to be on the same team/manager. They can just write black box tests to the spec, like the author of this post did.

I’m not even some big TDD guy. It just seems to me that in these core security-critical libraries/functions that should be pretty side-effect-free that you should have some basic “receive x, produce y” functional tests to make sure the API is doing what it claims to do on the tin.

[pvg]

the most basic of a test suite should be catching.

A most basic test suite is not likely to wait some arbitrary amount of time (2 seconds, as the author found by trial and error) between calls to the HSM.

[sneak]

The way I read the article was that it fails immediately, it just fails in a different way after two seconds.

Perhaps catching the instantly reproducible failure before release would have lead them to the root cause of both.

[pvg]

The images in the 'Digging deeper' section suggest otherwise. They appear to show a successful run followed by one that fails because the 'encrypted' value is garbage. Where am I missing the instantly reproducible failure?

[Bombthecat]

That's where fuzzing comes in.

[pvg]

In what way? 'Temporal' fuzzing to an eon-like range of two seconds seems, naively at least, entirely impractical.

Edit: a somewhat different way of putting this concretely - what is a practical stochastic testing regime that can reasonably be expected to find this bug?

[londons_explore]

Well designed systems have a mockable clock and timer subsystem.

They could easily test "Do X, wait 100 years, do Y".

You find all kinds of wired bugs when you do that - things that poll, for example Cron daily, will have to be run 36500 times. Certificates expire. Counters overflow. Date systems can't convert the date to and from strings. Logfiles get too big. Etc.

[pvg]

I guess if the answer to 'how do you fuzz time' is 'easily wait 100 years', I have no further questions.

[Bombthecat]

You know you can set the clock / time date of Systems without ntp? :)