[aiauthoritydev]

It is not crawling but indexing is the problem. Google has over years learned the patterns and authority of different articles. It will be hard for others to replicate but not impossible.

What Google should do is offer API based access to these providers but a lot of these providers might no adhere by contracts. So there is that.

[fooker]

Indexing is a fairly well understood technology.

You could hire one or experts for this to be doable with a pretty good amount of scalability.