Through sweat and blood. Don't expect crypto libraries to be secure the first few years after their release. That's why people favor the mature, battle-tested libraries.
Yes, but are hackers going after common used libraries to get more vulnerable systems to attack or are they going to spend time on some unknown homebuilt crypto? In some cases, security through obscurity works well in practice.
> In some cases, security through obscurity works well in practice.
This is not one of those cases. Absolutely not. I'm moderately competent at finding security bugs in things, but I doubt I could find any in OpenSSL. I am confident I could find some in your average hand-rolled code.
The thing is people make the same mistakes. There's a set of well-known mistakes that are very easy to make, especially if you're not versed with the entire history of implementing crypto - which is the case for the majority of people rolling their own. This makes it very, very easy to guess what mistakes they will make, and if you know what you're looking for it's easy to find it.
My "personal best" for finding a crypto bug in a project is 50 seconds.
I doubt anyone has found a bug in OpenSSL (or any established crypto project) anywhere near that fast.
You got to consider your most likely risk for attack. Targeted or at random by a botnet? For example, you are most likely more secure in practice by writing your own website than using Wordpress, simply because you are more likely to get hit by a botnet targeting every Wordpress site than someone going directly for you.
Both are vectors. On one hand, you have automated systems trying to hack low-hanging fruit (unpatched, well-known-to-be-insecure libraries), while simultaneously, you could become a specific target - where security by obscurity doesn't get you much. Saying "one of these is not worth considering, as it's less probable" is security through handwaving: one day, someone might discover a bug in your obscure library, and suddenly you're right back in range of automated attacks.
tbh, they do develop like this. Some guy come up with a library, a lot of critics arise, but the guy pushes through and many years later it becomes somehow a reference when someone big on twitter mentions it as his/her favorite crypto library.