[chrislomax]

Anyone else thinking they added a robots.txt entry for bing to disallow, waited to get de-listed then removed the offending robots.txt entry, posted this news article knowing bing would pick them back up in a couple of days?

[michaelt]

The HTTP last modified header for www.kogan.com/robots.txt has Last-Modified: Wed, 06 Jun 2012 06:14:35 GMT

It's possible they did this and they were cunning enough to turn back the last-modified clock.

[reinhardt]

How did you get that? Curl says May 4 instead:

  $ curl -I http://www.kogan.com/robots.txt
  HTTP/1.1 200 OK
  Server: nginx/1.0.14
  Date: Mon, 09 Jul 2012 18:48:34 GMT
  Content-Type: application/octet-stream
  Content-Length: 79
  Last-Modified: Fri, 04 May 2012 18:58:02 GMT
  Connection: keep-alive
  Accept-Ranges: bytes

[michaelt]

I get Last-Modified: Fri, 04 May 2012 18:58:02 GMT from my home machine (and you obviously do too) where I connect to 74.118.68.224 - but I got Wed, 06 Jun 2012 06:14:37 GMT from work (and via http://web-sniffer.net/ and http://www.rexswain.com/cgi-bin/httpview.cgi) which was looking at 108.166.39.66. Perhaps we get DNS load balanced or something similar?

[jbigelow76]

If that's what they did (no telling really) that would be pretty clever PR wise.

[rmc]

Clever… and dishonest.

[taligent]

Very much Kogan's style.