A belief that me flying would be safe(ish.) Mostly, I'm worried about fucking up in mid air and killing people. I don't know what the safety margins are for recovering from f-ups, so I don't do it.
When I was learning, I wasn't a safe driver. I learned to drive because of its obvious utility, but the only reason to learn to fly would be for enjoyment. The process of learning to fly with the prospect of killing myself, or someone else, would not likely be fun. I'd always have that image of screaming fiery death at the back of my mind.
It depends what the margins are though. If it is the case that if you mess up the instructor can recover the aircraft, then there's not too much to worry about – you're not going to kill someone while you're learning, or at least not with any high degree of probability. But not knowing the margins, not knowing how safe learning is.... That's the thing, rather than what would happen if I did get good at it.
Aviation is safe when pilots don't do stupid shit. Just like drivers killing themselves (and others) driving drunk or while texting.