A filter matches lines in the nginx log using regular expressions. If the line matches, it uses another regexp to extract the IP address, and then calls out to scripts to block that IP address.
I'm not going to post the exact configuration files I use, but the GitHub repo for fail2ban contains examples.