Hacker News new | ask | show | jobs
by j1elo 1520 days ago
Sadly very few authors seem to acknowledge or even know that github wiki pages are not indexed by search engines so if it wasn't for third-party sites like github-wiki-see.page (which could stop working at any time) their contents would be undiscoverable by the very same people they are usually intended...
1 comments

What? That's crazy! Does Github block indexing?
There's more details on https://github-wiki-see.page/ and https://github.com/github/feedback/discussions/4992#discussi...

> we have also introduced an x-robots-tag: none in the http response header of Wiki pages

> Abusive behavior in Wikis had a negative impact on our search engine ranking

> GitHub is currently permitting a select criteria of GitHub Wikis to be indexed

https://github.com/robots.txt

I don't see anything here about wiki specifically but maybe one of the rules hits wiki pages?

They've moved from robots.txt to blocking by headers.