Y
Hacker News
new
|
ask
|
show
|
jobs
by
rvz
82 days ago
> > Be sure to protect friendly bots and search engines from Miasma in your robots.txt!
Can't the LLMs just ignore or spoof their user agents anyway?
1 comments
phoronixrly
82 days ago
Well-behaved agents will obey robots.txt and not fall into the trap.
link