Hi, I actually work for Incapsula. For Layer 7 mitigation we use a multi-vector approach which' among other things, consists of:
Client Classification - comparing visitor's user-agent, IP, header parameters and etc to our pool of 10M signatures. Suspects will get CAPTCHA. (~0.01% false positives)
Visitor Reputation - we use crowd-sourcing to compile a list of suspected IPs. The list is updated in real time. Combined with other signals, this data allows us a better understanding of the incoming traffic.
Progressive Challenges - We check visitor's ability to retain cookies, execute JS and so on. In this case, the browser-based bots were able to evade those defenses. (These are also the most commonly used Layer 7 mitigation methods.)
Behavior Monitoring - We look at abnormal access rates, visiting patterns, etc. Here we also look for correctional of signals, to help us pinpoint suspicious behavior.
And so, by collecting and cross-referencing different types of data, the system is designed to distinguish between humans and bots. The process is mostly automated and is always seamless.
This is a great & informative answer. PhantomJS allows an attacker to get past any progressive challenge but it is nice that there is something else to go on. A properly-executed PhantomJS DDoS is a scary thing, it's great that you have some methods of mitigation.
I'd assume the HEADERs would be the same, e.g. basic stuff like UAS could be the same, or more low-level like Accept-* headers.
Note that this doesn't seem like a DDoS focused on high-bandwidth but rather focused on causing application load by triggering some server action and abusing the CPU/RAM, etc. In this case, Level7 analysis, is very effective and usually simple.
Very true. As far as headers concerned, we actually dig very deep. For instance, we will look at little encoding-related nuances, which can help identify spoofed headers (ua and IPs are fakeable, after all) :)
Also, we look for abnormalities in header order while being aware of variants that can derive from using various devices, proxies, etc.
Hence the 10M signature pool, which grows as new variants are spotted across our network.
Maybe they're using another form of pattern recognition. Time of day for the originating traffic vs. area of site being visited or particular type of stock etc...
Client Classification - comparing visitor's user-agent, IP, header parameters and etc to our pool of 10M signatures. Suspects will get CAPTCHA. (~0.01% false positives)
Visitor Reputation - we use crowd-sourcing to compile a list of suspected IPs. The list is updated in real time. Combined with other signals, this data allows us a better understanding of the incoming traffic.
Progressive Challenges - We check visitor's ability to retain cookies, execute JS and so on. In this case, the browser-based bots were able to evade those defenses. (These are also the most commonly used Layer 7 mitigation methods.)
Behavior Monitoring - We look at abnormal access rates, visiting patterns, etc. Here we also look for correctional of signals, to help us pinpoint suspicious behavior.
And so, by collecting and cross-referencing different types of data, the system is designed to distinguish between humans and bots. The process is mostly automated and is always seamless.