Kalman filters and more advanced aggregators add non-trivial latency. So even if one does not care about cost, there can be a drawback from having an extra sensor.
Cars and roads are built for human reaction times. That's why you have low speed limits in urban areas. You can have a pile of latencies attributable to processing a scene and still have superhuman reaction time that contributes to outperforming humans.
It's analogous to communications latency. High latencies are very annoying to humans, but below a threshold they stop mattering.
But also, if you didn’t get the right result, I don’t care how quickly you didn’t get it.