How sure are we about this? How certain are we that those specific species of mold have a net negative effect, rather than a net positive (like for example mushrooms)? Penicillium grows on stale foods and I doubt eating it would have a net negative effect.
Muchrooms have mycotoxins too. And red meat is a carcinogen. And predator fish have plenty of heavy metals. And the list goes on and on. Yet we eat all those things. Hence the "net positive".