[hpriebe]

Really interesting to hear that ProPublica used this sort of feature in a news context.

What other methods did you guys use to investigate "unpublicized" events/occurrences? What barriers did you face to finding that info in general?

[danso]

I'm no longer at PP so I can't speak for them (also didn't work on the aforementioned lobbyist story, but I admired the clever use of known data)...but I think invariably, every investigative story starts from traditional ways: a tip from a source on your beat or just plain research and knowledge of the domain. These newer techniques are just more domain-knowledge...lobbyist registration and activities can be found, in part, at House.gov. From that data, and records of Congressional travel, and patience to comb through PDFs, you can get more leads. If you mean engineering methods, such as machine-learning...I think the use of that is limited...not that people haven't tried (both in academia and in the news business) but because the datasets aren't organized or large enough to justify it.