[scarletham]

Take this with a grain of salt as I don't work at any of the funds mentioned (but still in the industry), but why not?

Filings are subject to change due to errors, updates, etc. It's cheaper, easier, and less error prone to just scrape and overwrite data than search for any updates.

I may have been heavyhanded in saying that the firms are rescraping the entire universe of filings on a daily basis, but I guess my point here is that these numbers can very easily be skewed and are generally a pretty poor indicator of actual data access.

[calabin]

SEC EDGAR is terrible for a variety of reasons. It breaks, few things are truly standardized/normalized, etc. If you have the time and an invested interest there's no reason not to.

Filings are subject to change but they are restated, not modified in place. That said, anything goes on the site.

The only real reason not to scrape everything on a continuous loop is that there is a rate limiter in place, so you may end up impeding your intake of new data by loading yourself down with old jobs to handle.