[readerrrr]

I wouldn't worry myself with that. Since a black hole behaves exactly like a normal star at a large distance, the chance is very small, even if there are many of them. For example, when our galaxy merges with Andromeda there will be only ~1 star collision.

Normal black holes are really small, much smaller than a planet, which we can barely detect now. And those planets must be orbiting a star and in the correct inclination to be detected.

[BrianKoberlein]

Stellar mass black holes are small in size, but supermassive black holes can be larger than our solar system. Fortunately for us they only exist in the centers of galaxies. The closest supermassive black hole to us is 30,000 light years away.

[Tloewald]

Also even small (stellar) black holes are very massive — 5x sol and up — (since they tend to occur from the collapse of very large stars) so having one pass near or through the solar system could be catastrophic.

Primordial black holes are posited to be possibly so small as to be able to pass through the earth without interacting with more than a few atoms, which would make visiting them both difficult and not enormously interesting.

Edit: clarification and added info on primordial black holes which I looked up.