[cryptarch]

I feel like having "safety net" sections is a smell, too. Maybe it's due to limitations of the language? I'm not a C expert.

Safety nets make it seem like you're handling edge cases in a vaguely specified, ad-hoc way, which is prone to forgetting to add the safety net in at least some places, while the nets themselves are easy to mess up as well.

Could this code benefit from more typing perhaps? Automated checking of pre- and postconditions? Are there C (macro) libraries implement that in a usable way?

[jeltz]

Yeah, the real smell here is the "safety net" which is also what seems to have caused the bug. Clean quality code should avoid this kind of safety nets as much as possible and instead make it hard in the first place to get the program into an invalid state and if we do get into an invalid state not just silently try to guess what the user really wanted to do. As it turns out people actually wanted to return success with an error code and relied on this being possible in their applications.

A better type system would help out a lot here, but it is also possible to write clean C code without this class of bugs. And OpenSSL has some of the worst code I have seen in an open source project (I have seen much worse in commercial projects), so while C has a lot of flaws do not judge it after OpenSSL.

[kroeckx]

I think there is a misunderstand of what the existing safety net is about. There are 2 error states: did the verification fail and should the connection be aborted. The safety net makes sure that if a function (the callback) says the connection must be aborted but didn't set the verification error, that it sets an unknown verification error. Note that the callback is external code.

The new "safety net" that libressl added said that if the connection doesn't need to be aborted there was no verification error.

[dom0]

"safety net" cases are pretty much covered by design-by-contract 101.