[_yb2s]

This article addresses that. They claim that, like you said, they are only used at very high altitude or in space, and the raw materials, before ignition are not dangerous. They say that the low enrichment uranium is even safe to touch with your bare hands.