We have a platform running on more than 50 servers which includes crawlers, website categorizers, entity extractors, etc. It scans about 30 millions websites (categorized as corporate websites with content in English) on a regular base. The goal is to build a structured profile for every known company website.. and repeat this process every month in order to compare historical profile and generate "transactions" like "company X changed CEO", etc...