It's not just a nagging doubt, it's adding a "gap" to your resume - I've heard many employers view any sort of exclusive self-employment as basically that and it's something that actively keeps me hesitating.
I did it, and failed, and the gap year only hurt economically. My job prospects were better if anything after (and changing jobs netted a large enough raise that it soon made up for the gap year.) I do regret that I failed, but I think I needed to take the shot. I'm considering doing it again.
I see this excuse a lot. Maybe it’s a valid reason in some industries. But IMHO it’s likely just a convenient excuse someone plays to further procrastinate on making the though decision now so they don’t have to make a tougher, maybe-very-unlikely decision down the road if they want to re-enter the workforce.