[btilly]

There could well be millions or billions of sand particle sized objects coming through constantly.

The Solar System is big. Almost all of them will pass through and miss everything. We literally have no way to notice them.

Incidentally we have spotted a second interstellar object. https://en.wikipedia.org/wiki/2I/Borisov is more what we were expecting them to look like.