Automating Scrapers

[hrishipartha]

with so many new developments in AI, i was wondering if there are easy ways to create automated trackers (based on public news, internet) using a set of criteria.

the idea of the final product is simple. a running tracker of all events that meet a particular criteria, with hyperlinks to source references. the use case could vary - from road accidents to other kinds of events that people are interested in, e.g. crimes of a particular kind. sometimes the source events could be public internet, in other cases documents that could be uploaded to the software (such as death certificate records etc.)

folks should consider the utility in a country/context where information is not freely available or published transparently.

[easytiger]

AFAIK the LMs are not continually updated though?

[datathrow0007]

Google Alerts?

[hrishipartha]

Google alerts is too simplistic, one needs a deduplicated tracker of events, an ability to scrape from public records as well as news sites

Perhaps it can be a starting point, but some sort of manual/automated natural language processing is needed on top