Hacker News new | ask | show | jobs
by olivierkaisin 4135 days ago
I wouldn't be surprised to hear that they did put that url into their robots.txt because it is a search result page generated by a bot and that has been visited and indexed by Google…
1 comments

I'm not up to date on how google's current search indexing algorithm works. Supposing someone has a lot of bots that post links to `extra-small-teen-pony` or some other /sr query on nordstrom.com from other sites - would google index that?
If they threw a custom 404 that reports 200. That's how I assumed those phone number lookup sites work.