|
|
|
|
|
by jsheard
624 days ago
|
|
https://www.404media.co/google-is-the-only-search-engine-tha... https://www.reddit.com/robots.txt Reddit serves a different robots.txt to Googlebot, you can see a snippet of it in Googles summary. If Kagi is getting recent Reddit results then they must be either ignoring robots.txt, using Google as a backend, or also paying Reddit for access like Google. Bing and DuckDuckGo are certainly still locked out, I just tried searching for "reddit hurricane milton" on both and none of the results are actually from Reddit. |
|
https://help.kagi.com/kagi/search-details/search-sources.htm...