Hacker News new | ask | show | jobs
by Philpax 329 days ago
Yes, they're uncompressed. For reference, `enwiki-20250620-pages-articles-multistream.xml.bz2` is 25,176,364,573 bytes; you could get that lower with better compression. You can do partial reads from multistream bz2, though, which is handy.
1 comments

Kiwix (what the author used) uses "zim" files, which are compressed. I don't know where the difference come from, but Kiwix is a website image, which may include some things the raw Wikipedia dump doesn't.

And 57 GB to 25 GB would be pretty bad compression. You can expect a compression ratio of at least 3 on natural English text.