[humdumb]

I'd say this is a flaw of the "web" and relying on hyperlinking and "HTML forms" as a means to access all content. That's what web crawlers do.

Better would be putting databases on certain known ports, with no "hostname" requirements and let search engines scan IP ranges for openings on those ports. You send a ping to the port and if a "Welcome" response comes back, then you have a public database to explore. The response might even describe the contents of the database. (Yikes, I'm having flashbacks to gopher.)

What's funny is that "port scanning" is like the term "hacker" it has a negative connotation even though there's nothing inherently devious about it. Pinging a port to detect if there's a public database open on it is actually a more efficient less resource intensive means of service discovery than crawling hyperlinks in HTML generated by some web developer (maybe even one employed by a governemnt agency with a tight budget) hoping to stumble upon every possible data resource.