[rabidrat]

I use local tools for all my data processing, and if I want to search for a relevant dataset, I'd like to do it locally instead of having to contact an API and do all the paging etc. The kinds of queries that might be important to me are outside the scope of any fixed API; I might want all datasets updated in December (of any year), or only those with between 100k and 1m rows, or to get a frequency table of datasets by license terms. It's very easy to do any of these queries if I can download a TSV (or equivalent) with all the metadata, and usually too frustrating to even attempt if I can't.

[kevinemoore]

Maybe the Elasticsearch cluster would help you here. Because all the datasets (and files within datasets) are indexed along with their metadata, you could write elastic queries to find the datasets you want--as long as the dataset creators are including the relevant metadata in the dataset annotations.