Hacker News new | ask | show | jobs
by oblio 1511 days ago
What? That's crazy! Does Github block indexing?
2 comments
bckygldstn 1511 days ago
There's more details on https://github-wiki-see.page/ and https://github.com/github/feedback/discussions/4992#discussi...

> we have also introduced an x-robots-tag: none in the http response header of Wiki pages

> Abusive behavior in Wikis had a negative impact on our search engine ranking

> GitHub is currently permitting a select criteria of GitHub Wikis to be indexed

link
beembeem 1511 days ago
https://github.com/robots.txt

I don't see anything here about wiki specifically but maybe one of the rules hits wiki pages?

link
crazysim 1505 days ago
They've moved from robots.txt to blocking by headers.
link