[velshin]

Googling it says limit user processes via /etc/security/limits.conf http://goo.gl/RJJhN

Is it not that simple?

[tzs]

I believe that for each level of the challenge, all users were ssh'ing in to the same user account, so even with per-user process limits a fork bomb would use up all the processes available to the competitors at that level.