No, many fact check websites (such as Snopes) will debunk conspiracy theories by simply saying "there is no evidence to support the theory, it's just wild speculation."
It does constitute a valid fact check. It is ridiculous to claim otherwise.
Claim: EdiX is a crazy pedophile
Fact-check: This is just random speculation and there is no evidence.
By your standards an actual fact-check would need to cite testimony from your (ex-)partner(s)/spouse(s), a psychological evaluation, maybe some videos of you not being a pedophile around children. Utterly ridiculous.