[lorean_victor]

it was quite easy for a small team to crawl and index a good portion of the internet, enough to become the de facto gateway (talking about Google).

it was similarly possible for a relatively small team to crawl a good chunk of the available internet and train some of the most sophisticated "algorithms" we've seen on them (talking about Open AI).

if there is an incentive, this problem can be solved. if this was actually a hard problem, most current social media companies wouldn't put so much effort in restricting crawling to force everyone through restricted API access (look at Twitter, Reddit, Instagram or Facebook, as examples).