Hacker News new | ask | show | jobs
by blauwbilgorgel 5093 days ago
They serve their robots.txt as a "application/octet-stream" not "text/plain". Might be confusing some search engines?

Also according to Internet Archive their robots.txt has changed in the last months.

This was a version that was archived:

http://web.archive.org/web/20080104044407/http://www.kogan.c...

At a glance, they are whitelisting some bots, including Googlebot, but not including Bingbot, and disallowing the rest.

  User-agent: Googlebot
  Disallow: 

  User-agent: *
  Disallow: /
Another marketing stunt?
1 comments
UnoriginalGuy 5093 days ago
Your link is pointing at 2008. That is four years ago. Hardly proof of anything.

You are right about the content type however.

link
blauwbilgorgel 5093 days ago
Correct. It doesn't proof much, but that they previously had a buggy robots.txt.

Usually these posts are accompanied by a declining traffic graph for Bing referrers. That would give us more proof and a timeframe to work with.

The post gives two options for their absence in Bing results: a fault at Microsoft or as a punishment. With their history I think a fault on their own part is a more likely option. I can't rule that out.

link
Yver 5092 days ago
You are arguing in bad faith. The robots.txt from 2008 is a whitelist of 190+ known crawlers. It doesn't include Bing, which was launched in 2009.

In 2008, kogan.com was a parked domain. It has nothing to do with the current owner(s).

link
blauwbilgorgel 5092 days ago
The choice for linking to a 2008 file was not the best. At least their current robots.txt is buggy too, so it isn't all that relevant for proving a track record.

I thought kogan.com was around in 2008 as the blog goes back to 2008.

The robots.txt is whitelisting bots with agents like "No" and "Due to a deficiency in Java it's not currently possible to set the User-agent." but then blocks all other known crawlers (like MSNbot and Yahoo Slurp).

I was not trying to be deceitful. IMO: Deceitful is a PR stunt arguing it is the fault or an evil plan of Microsoft, when you don't present anything to substantiate your claim, and it is a fact the webmaster is at fault in the vast majority of these cases.

link