Unless your or your kids immune system is compromised, that's probably a bit too much care (we did go to similar lengths with a preemie, but not with our second kid).
Yes, for dirt I wouldn't worry at all... but e.g. rodent excrement is quite dangerous: leptospirosis (kidney damage, meningitis, liver failure), hantavirus, salmonella, LCMV. I know, chances are low but I'd be happy to reduce them as much as possible.