Hacker News new | ask | show | jobs
by rvz 82 days ago
> > Be sure to protect friendly bots and search engines from Miasma in your robots.txt!

Can't the LLMs just ignore or spoof their user agents anyway?

1 comments

Well-behaved agents will obey robots.txt and not fall into the trap.