You don't need a license or formal training, but the reason for that is not because there is some right to fly ultralights. It is because the FAA rules specifically give permission to fly ultalights without a license or formal training. See 14 CFR 103.7 [1].
Actually, that's a very good point. There are some important restrictions regarding airspace but there are legal ways to get up in the air by yourself without any training
The willingness to risk your own life in an ultralight is enough to satisfy the FAA that you followed some minimal due diligence; however, you're not risking anything by flying a drone except a few dollars.
[1] https://www.law.cornell.edu/cfr/text/14/103.7