Y
Hacker News
new
|
ask
|
show
|
jobs
by
ggirelli
205 days ago
Out of curiosity, I see my website not being well indexed and I wondered whether it is because I include ``` User-agent: * Disallow: / ``` in my robots.txt. What should one add to allow Marginalia to crawl their website?
1 comments
marginalia_nu
205 days ago
This should work
User-agent: search.marginalia.nu Allow: /
link
ggirelli
200 days ago
Thank you! I just added it :)
link