[fragmede]

Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.

[ccgreg]

Common Crawl's dataset was downloaded in full 100 times in 2025.

We agree that it would be great if it was even more widely used.

[dylan604]

as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?

[ccgreg]

Common Crawl's archive has metadata that says when each record (html file) was crawled.

[charcircuit]

But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.

[ccgreg]

We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.

[charcircuit]

Sorry, I wasn't clear. In order to know you have an up to date copy of the page you need two pieces of information:

1. When the page itself was last updated

2. When the crawled copy was last updated

In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.

The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.