Hacker News new | ask | show | jobs
by ggirelli 205 days ago
Out of curiosity, I see my website not being well indexed and I wondered whether it is because I include ``` User-agent: * Disallow: / ``` in my robots.txt. What should one add to allow Marginalia to crawl their website?
1 comments

This should work

  User-agent: search.marginalia.nu
  Allow: /
Thank you! I just added it :)