[mmh0000]

Something like this can be implemented in Linux's NetFilter quite easily using the recent module:

    iptables -A FORWARD -i eth0 -m recent --update --seconds 600 --hitcount 10 --name scan --mask 255.255.255.255 --rsource -j DROP
    iptables -A FORWARD -d 173.165.141.72/29 -i vmexfwbr -m recent --update --seconds 86400 --name instaban --mask 255.255.255.255 --rsource -j DROP
    …PUT STANDARD ACCEPT/REJECT RULES HERE…
    iptables -A FORWARD -d 173.165.141.72/29 -i vmexfwbr -p tcp -m multiport --dports 21,23,110,1433,3389,5060,8080,8088 -m recent --mask 255.255.255.0 --set --name instaban --rsource -m comment --comment "Instabans port scanning bots."
    iptables -A FORWARD -d $YOUR_IP_RANGE/29 -i eth0 -m recent --mask 255.255.255.0 --set --name scan --rsource
    iptables -A FORWARD -j REJECT --reject-with icmp-port-unreachable

The above would start dropping packets from anyone who hits 10 or more ports not previously accepted by an accept rule, and block the /24 of anyone who hits one of the more commonly scanned ports.

[LinuxBender]

Use care when blocking scans with iptables. This is mostly safe for TCP if you have a dummy socket listener for the attacker to connect to and you look for established connections in iptables. If you block on SYN or UDP packets then I can disable your traffic with simple spoofing. I can even get you to block your own gateway unless you whitelist it.

[jimsi]

This might be a way better solution in terms of performance, but I cannot imagine way to run more complicated stuff rather only blocking IP in the firewall using iptables