Only if they’re uncrewed. Any idiot can legally fly an ultralight experimental aircraft with basically zero oversight. I don’t think you even need a pilot’s license. But God forbid you try to do the same thing without putting a pilot at risk; that’s effectively illegal.
That makes some amount of sense. If you put the ultralight pilot in the craft, they literally have skin in the game. Further, ultralights cannot be flown over “congested areas” of a town/city.
Trying to apply those rules to drone deliveries while ignoring those two pretty fundamental elements doesn’t strike me as evidence that the FAA is being arbitrary or capricious.
When you're risking your own life, you'll likely be at least somewhat careful.
When you're just flying an empty test vehicle you'll be much more reckless, and thus way more likely to crash it on other people's property or other people.
Part 103 aircraft don’t need a license of any sort. There are of course a bunch of regulations about how/when/where you fly it, but you don’t need any official training or certification.