Hacker News new | ask | show | jobs
by gavinhking 817 days ago
I made this, let me know if you have questions or feedback.
1 comments
tbeseda 817 days ago
Thanks for the work on this!

I automated my site's robots.txt[0] by scraping your site. It would be extra nice if darkvisitor.com exposed a plain text version or JSON representation of the list.

[0] https://tbeseda.com/blog/automating-my-robots-txt-to-block-a...

link
glynnormington 817 days ago
That was one of the points of the new repo. A plain text version of the file is https://raw.githubusercontent.com/ai-robots-txt/ai.robots.tx...

The other point was to make this community maintained rather than rely on one source to provide all the inputs.

link
tbeseda 817 days ago
Definitely! And I'll likely use that raw URL in my tooling going forward. So thanks for starting the repo.

I'm thinking it also helps to bring up a feature request on the source material so we can all limit the drift. ie. if darkvisitors.com had a sort of plain text API, your repo could check for new entries via GH Actions and create issues or even PRs.

link