[kauegimenes]

https://www.reddit.com/robots.txt replies Not Found over here

[mzmcbride]

This URL was returning "Not Found" for me as well, but with an HTTP 501 status instead of the traditional HTTP 404. After investigating, the issue seems to be cookie-related:

  $ curl -sIXGET 'https://www.reddit.com/robots.txt' -H 'cookie: rseor3=true;' | head -1
  HTTP/2 501

Compare with:

  $ curl -sIXGET 'https://www.reddit.com/robots.txt' | head -1
  HTTP/2 200

[kauegimenes]

Thanks, works from incognito mode in browsers also then :)