[wongarsu]

> but I don't think the conflict of interest would allow for cost-effective large-scale access and pricing

Not for existing search machine providers, but I think there is room for new players to do this large scale. Imagine an AWS service that high performance access to crawled data as well as a number of indexes and a fairly simple search engine using this data. That would commoditize one of Google's biggest advantages, and anyone could, at least in principle, run their own search engine from the data. Because the market for this is much wider than traditional search engines just providing the data and indices for a pay-as-you-go fee could still be very profitable.