Y
Hacker News
new
|
ask
|
show
|
jobs
by
hundchenkatze
311 days ago
You left out the part that says Perplexity-User generally ignores robots.txt because it's used for user requested actions.
> Since a user requested the fetch, this fetcher generally ignores robots.txt rules.