It could very well be but I am very close to some of them and am confident that they are neither oblivious nor feeling like they have to hide anything. They are well respected in their respective communities.
Your anecdotes do not support your stated conclusion that "it is a tiny minority"; at best, even if your data is actually accurate, you've gathered evidence to support the conclusion that there exist people who have not been harassed or abused. Likewise, there is more than enough evidence to support the conclusion that there exist people who have been, and more than enough evidence to infer that those aren't just "isolated incidents" (http://geekfeminism.wikia.com/wiki/Isolated_incident).