Hacker News new | ask | show | jobs
by jaybna 537 days ago
Yeah, probably right. If you want a great rabbit hole, look up "Common Crawl" and see how a great academic project was absolutely hijacked for pennies on the dollar to grab training data - the foundation for every LLM out there right now.
1 comments
CamperBob2 537 days ago
It's hard to envision a greater success for the "great academic project" than what happened. I mean, what else were they trying to accomplish?
link
jaybna 536 days ago
It was meant to be an open-source compilation of the crawled internet so that research could be done on web search given how opaque Google's process is. It was NOT meant to be a cheap source of data for for-profit LLM's to train on.

*edit: added "for-profit"

link
CamperBob2 536 days ago
(Shrug) Multiple not-for-profit LLMs have trained on it as well.

If something I worked on turned out to play a significant part in something that turned out to be that big a deal, I'd be OK with it. And nobody's stopping people from doing web-search studies with it, to this day.

link