[Kudos]

Speaking as the operator of a similarly themed site, allowing drive-by downloads is a huge target for malware uploaders.

[ogdr]

So how do we solve this? By only allowing hotlinks on Images or similar?

[Kudos]

That's precisely what I do. You probably also want to scan the files for viruses at the time of upload and at regular intervals. I use ClamAV and F-Prot.