Well, most of the other theories back then would seem ridiculous and obviously wrong nowadays. That's the nature of theorizing when you have little information.
Obviously we mostly remember those theories that turned out to be right.
We're not sure it's matter, but let's just say 'dark matter' is a useful abstract label (like 'teapot'). We know that SOMETHING is perturbing our models that are useful (to simplify the analogy, let's just posit that newtonian orbits are casually affected, for example) by measurement. As much as an orbiting 'teapot' is ridiculous, it's also provable that something is unaccounted for, 'teapot' or 'dark matter', whatever the label.
This does not mean that it's matter specifically (just a label), but is a phenomena.
Define "directly". The definition of dark matter is that it's electromagnetically non-interacting, therefore observing it's gravitational effect is as direct as it gets.
If you cannot detect something, it's can be equally ridiculous to say it exists.