Y
Hacker News
new
|
ask
|
show
|
jobs
How did OpenAI crawl the web so effectively without getting blocked?
6 points
by
BrownSol
1053 days ago
4 comments
cratermoon
1053 days ago
It didn't. Others did it, OpenAI used it.
https://commoncrawl.org/
link
throwawayadvsec
1053 days ago
I'm pretty sure scraping is easier than building a groundbreaking LLM used by 100M people
link
JoeyBananas
1053 days ago
It's not hard to scrape the internet
link
arthurcolle
1053 days ago
Very carefully
link