[Gravityloss]

There are relatively simple ramjets that fly at around Mach 4 speeds. They are unmanned and are accelerated to speed by solid rockets. Probably the ramjet can't even start below Mach 1 or so.

Designing an air breathing propulsion system for a wide speed range is difficult.

Rockets are much easier after a certain point. McDonnell Douglas and Paul Czysz had interesting projects. If you want manned reconnaissance, a Mach 6 air launched liquid rocket powered lifting body would probably be the next logical step from the Blackbird and would not even be super hard. With rockets you don't have the inlet problem at all and they have excellent thrust to weight ratio.