Hacker News new | ask | show | jobs
by SR2Z 521 days ago
IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.
2 comments
prinny_ 521 days ago
Isn’t this a weak argument? OpenAI could also say their goal is to learn everything, feed it to AI, advance humanity etc etc.
link
compootr 521 days ago
OAI is using others' work to resell it in models. IA uses it to presrrve the history of the web

there is a case to be made about the value of the traffic you'll get from oai search though...

link
SR2Z 515 days ago
It does depend a lot on how you feel about IA's integrity :P
link
amarcheschi 521 days ago
I also don't think they hit servers repeatedly so much
link