This is why we use a redundant geometric approach (with LiDAR) as a fallback to NNs in our perception stack. NNs can have false negatives which is unacceptable, but dealing with false positives is more manageable and safer.
Since a lidar uses a discrete frequency band and it's own energy source, it's much easier to process the signal for depth information (using the lag between sending and receiving a return pulse(s) and the wavelength). Arguably such a system might provide a better basis for a backup system based on modelling obstacles and their distances from the vehicle.