Hacker News new | ask | show | jobs
by tim_iles 5499 days ago
Last time I downloaded Wikipedia, it was 4.5GB. If I were to knock up this hack, I would definitely scrape pages instead.
1 comments

On-demand page scrape + memoisation is almost certainly a win here. Even if thousands of people are hitting this, a lot will choose some of the same queries (I'm sure Kevin Bacon and xkcd and philosophy are in there a bunch), especially in the tails of the paths (Latin, Mathematics, ...)