[crdb]

I think it's simpler than that: there's a war on between the SEO lot and Google. As SEO teams get better and better at mimicking a signal, Google needs to keep upping their game to let the valuable stuff get through without being drowned in about.com spam (sometimes the same text can be found in 10 successive clones of about.com) or companies upping their ranking. Then you get feedback loops as people's expectations adjust to the new reality.

Right now, if I type "!g clothes" my top result is missguided.co.uk; if I go to google.com and type "clothes", my top result is nastygal.com (ironic that both of these companies have "naughty" branding, when they are so good at spamming search results). Clearly both of these fashion companies have excellent SEO teams, but there might also be a factor of people expecting shops when they type a non-brand keyword.

DDG is also filled with shops (presumably affiliate links, since that's their business model) but the top/separate result is the Wikipedia article for clothing, which is actually informative if I wanted to read about clothes.

It's a good business decision for Google to have high quality search results. The problem with the previous search companies is that they sucked. Most/all of the top results were paid. There was little thought about relevance. Google won so hard because it allowed people a way to browse the enormous amount of information on the web in a structured way taking into account relevance. The ads need people to keep coming, which won't happen if other search companies are more relevant. The dynamic will be that whoever wins the search battle will end up being the new target for SEO spam and eventually develop the same problems.