[phillmv]

You seem to have a lot of experience with this kind of stuff.

Could you possibly give me a tldr on this story?

I recall the exposé talking about how they were trading twitter links for keyword link building. Yet I'm having a hard time working up the requisite outrage.

It's not spam - the people linking back aren't being coerced (i.e. spam comments), and the content it's being linked from is legit (i.e. not a crappy link-farm site void of content).

Is the reason that this is Bad(tm) because the link back is not "organic"? It strikes me as being identical to say, paying every a thousand bloggers writing about Bieber to link to RG - except that last example is impossible to detect. They exchanged a small ad for twitter inbound traffic.

Is there just a blanket ban on trying to divine how the algorithm works? It's a commonly accepted practice to pay other people to promote or write about your product/service/brand.

[itsprofitbaron]

Essentially Rap Genius violated one of Google's Webmaster Guidelines[1] by attempting to manipulate the SERPs through getting webmasters to link to several lyrics pages in exchange for a tweet. In doing so, Google considers this as link scheme[2] which is trying to manipulate the results specifically in relation to Buying/Selling Links:

  Buying or selling links that pass PageRank. This includes exchanging money for links, or posts that contain links; exchanging goods or services for links; or sending someone a “free” product in exchange for them writing about it and including a link

There are several other ways to "manipulate" the SERPs this way including some of them which you have identified such as - spam comments, doorway pages, forum profiles amongst others - and Google has an algorithm codenamed Penguin which, detects and penalises webmasters who attempt to manipulate the search engines in such way (although it is more complicated than this).

However, Penguin is not the only way which Google identifies people being involved in these practices as they also have a place to report the links[3].

This is one of Google's Manual Actions[4] that webmasters receive, when Google believes you are not providing the user with additional value and/or are trying to manipulate the results.

They cover everything from Thin Content (mainly through Panda) to Hacked Sites to User Generated Spam to the recent Image Mismatch Penalty etc. You can see them all here: https://support.google.com/webmasters/topic/2604771?hl=en&re...

[1] https://support.google.com/webmasters/answer/35769

[2] https://support.google.com/webmasters/answer/66356

[3] https://www.google.com/webmasters/tools/paidlinks?pli=1&hl=e...

[4] https://support.google.com/webmasters/answer/2604824?hl=en

[usaphp]

> ...by attempting to manipulate the SERPs through getting webmasters to link to several lyrics pages in exchange for a tweet.

How is that different from paying bloggers to write a product review with a link to a product? Why is not that considered a "manipulation of SERPs" - you exchange (money/tweet) for a link.

[itsprofitbaron]

It's not different. As mentioned earlier, paying for links is against one of Google's Webmaster Guidelines[1] and your example specifically falls under the Link Scheme[2] category which is affecting Rap Genius.

Those types of links reviews for a link, are considered advertorials which Interflora were "famously" penalised[3] for and something Google specifically identifies within their Link Scheme examples[2]

[1] https://support.google.com/webmasters/answer/35769

[2] https://support.google.com/webmasters/answer/66356

[3] http://searchengineland.com/google-says-no-comment-on-why-in...

[phillmv]

Wait.

I was about to ask if the lesson to draw here was to make it look organic.

But mostly serious question here: how does TechCrunch have any Google rank then? How do most tech publications, for that matter? PR hits are extremely common; they're the name of the game when it comes to cheap content.

It's okay if it looks like it's mutual self interest, and not if it overtly competes with Google's advertising platform?

[itsprofitbaron]

how does TechCrunch have any Google rank then? How do most tech publications, for that matter?

TechCrunch does not post advertorials, they post content for free and they actually posted on their site to reaffirm this fact[1].

Sure there may be some PR firms etc who might get paid to get that content on to TechCrunch or another tech publications but TechCrunch writers are not directly compensated for doing so.

As a result the content they write from a startup launch to a new feature etc is considered "natural" by the search engines as they're choosing to write about it.

Moreover, those publications get the majority of their search engine traffic through being in Google News ala. posting about "Twitter" and getting inserted into the "Twitter" SERPs within the "News" section. Additionally they also leverage internal linking to boost their SERPs potential e.g. whenever they talk about Zulily (they're in #9 for me in Incognito mode although they might be higher/ on Page 2 for you) instead of linking to the site they'll reference the "Tag URL"[2] as well as referencing Crunchbase[3] (although CrunchBase isn't really 'internal' as its an external site). Likewise, Google loves "fresh content" so they will naturally be inserted with little/no links within the top 7ish results for something generic (although this does not always happen) such as "credit card numbers" when they do a post about "credit card numbers" and will naturally lose search engine positioning for that term over time.

[1] http://techcrunch.com/2012/11/08/we-are-worth-at-least-3k/

[2] http://techcrunch.com/tag/zulily/

[3] http://www.crunchbase.com/company/zulily

[RBerenguel]

It's not different, and Google will penalise as needed (if the site is unrelated to the reviewed object, for instance, also the FTC can sue if there is no proper disclaimer)

[tehwebguy]

Interesting, does that make sponsored posts necessarily against TOS?

[itsprofitbaron]

Sponsored Posts are what Google calls advertorials and considered to be a link scheme[1] which means they're against their webmaster guidelines[2].

However, they're acceptable if they do not carry any link equity (aka. use the no follow tag) on the link.

[1] https://support.google.com/webmasters/answer/66356

[2] https://support.google.com/webmasters/answer/35769?hl=en

[graeme]

Interesting. I am in a small niche, where a lot of student society websites ending in .edu have links to their sponsors, and they don't nofollow them.

Typically they don't know what nofollow is, and their sites are run by people who don't really know much about websites.

I can't educate the entire sector. Is it better to steer clear of link building with this broad swath of sites? Most of the other sites in my niche have built links through these sites.

There is a shortage of followed links in my niche as most content creators in my narrow field are commercial and don't link to competitors.

[sharkweek]

Nah -- it's just like buying ad space on a site though, so the links to your site should be nofollowed.

[rsync]

"You seem to have a lot of experience with this kind of stuff."

Which is sad.

SEO is a cancer and, like cancer, there is no good SEO. Google and their broken search ecosystem have spawned a vast, useless, derivative "industry" that needs to die.

I say "broken ecosystem" because SEO should have a value that is inverse to how well google indexes. That SEO exists at all shows the extent to which it is broken. Website death penalties (or whatever we call this) don't solve the problem.

[vidyesh]

You are somewhat right and somewhat wrong.

SEO indeed is like what you described, but there is a fine line to start doing blackhat or stay whitehat.

Now the problem is, the line is so thin you end up crossing it at times. And everybody does. The way Google works you need to do both somewhat in a way to get the balance and stay top.

For example, link-building its one of the worst areas where defining white-hat and black-hat is really difficult. One way is doing guest post is allowed and great but paid posts aren't. There is no guarantee a guest post is not a paid post.

There are 'digital marketing agencies' which at times are so ridiculous that the backlink included in the guest post doesn't have any value in to the post as such but anchor text magic does all the work.

SEO isn't broken as such but there are too many backdoors to doing the white-hat SEO methods in the concealed black-hat way that it isn't realistic to blame he search ecosystem.

I agree, website penalties is not a good solution to the problem.

[blauwbilgorgel]

There certainly is good SEO. Good SEO is often indistinguishable from good accessibility, good marketing, good design, good conversion optimization, good information theory, good metrics.

You use sites because they used good SEO. You can find StackOverflow posts ranking at one, because they used the good SEO that the good content deserved.

Anyone with an internet business taking your comment at face value will find herself ranking not in the top 20, no matter how good their content.

Good SEO has worth, has value. Even Matt Cutts promotes good SEO and sees the worth. Without SEO analytics you are sailing blind. Without knowledge of how to interpret them. Without knowledge of the guidelines you get gaffs like these.

There is good SEO and there is vandalization. Rap Genius vandalized their rankings.

No industry needs to die. If it needs to, explain why.

Optimize your site for all users, not just search engines.

If you stay in this naive view of SEO you will never have a successful internet business in many niches. What is sad is that some developers and designers can not deliver a website that is SEO optimized, so a business owner has to pay double to get an SEO to fix all on-page accessibility issues.

This penalty is entirely the result of that blogpost and the PR it generated. I am willing to bet that Google already discovered many of these links, through spam reports, through manual spam fighters and through algorithmic detection. This PR, and the open letter, forced Google's hand. Of course they have to visibly act on it. Of course they are well aware that that niche has some scummy shady practices, and Rap Genius is not the only one. Of course they have a better internal tool than OSE and can check link profiles and detect networks in a manner of minutes.

Besides, you can be a great SEO and never do a single link-building campaign. There is tremendous value in simply finding new content and articles to rank for and drive traffic. I loath black hat SEO, not for the unfair temporary advantage it gives, but because it creates posts like yours. Because it burns companies that are now much more careful to hire good SEO's. Because I know that good SEO helps people access relevant content.

Like e-mail spam, search engine spam will die. SEO's will be retrained to create websites that are accessible, user friendly, give content that is relevant to the search query.

Just read: http://static.googleusercontent.com/media/www.google.com/en/...

And try to find a single practice in there that is either manipulative, entirely done to service search engines alone, or hampers accessibility and user friendliness.

Good SEO builds authority. Creates quality mark-up. Makes descriptive website titles. Implements breadcrumbs and a sane website hierarchy. Rids the index of duplicates. Drives more users to your sites. Makes sure that blind people can access your content, like images or video. Makes sites load faster. Makes content more trustworthy etc. Good SEO needs no manipulation. There is a lot to gain by simply aligning yourself with Google's vision.

Please reconsider your tone, or realize you do not add anything of value to the debate, or to the many online business owners on HN. Can you imagine raising funds for an online startup and then have your post be the slide for "online marketing"?

[rsync]

"Anyone with an internet business taking your comment at face value will find herself ranking not in the top 20, no matter how good their content."

... which is a direct indictment of google and their (as you put it) ineffective search results. If the best content is not being served based on its merit, the search results are bad.

A very good explanation for this is that optimizing for good search results and optimizing for high ad clicks from searchers are two different things, and you can't do both. Therefore the price of optimizing for search-ad-revenue is less than optimal results and the unintended consequence is a side-game that parasites play called "SEO". I mean parasite in the nicest possible sense and I think it's an apt description.

"Please reconsider your tone, or realize you do not add anything of value to the debate, or to the many online business owners on HN."

I'm not here for the business owners - I came for the hackers.

[blauwbilgorgel]

>which is a direct indictment of google and their (as you put it) ineffective search results

I think Google has the best quality index of all search engines. There is a Kaggle challenge for Yandex right now where you can optimize the search results taking into account data like user sessions and dwell time. It's hard, but very cool. Google owns hard and cool stuff. Web spam is a multi-faceted and complex problem. But these guys are training neural networks on Youtube stills and make it detect cats. The last time I found a top 10 spam result that irked me was months ago. I filed a report and move on to the otherwise great index.

>If the best content is not being served based on its merit, the search results are bad.

If the best content is in an image without an alt attribute or longdesc or HTML fallback, on a page with no surrounding text, no sourcing, no pagetitle, no meta description. On a domain that accidentally blocks search engines with robots.txt, has no structural interlinking of pages, no backlinks, takes 50 seconds to load, redirects crawlers to a different page by IP, and only works with javascript on. If that happens to be the best content, then that is a shame. Google could probably still index it :). But the search results are better for not ranking that inaccessible, untrustworthy, undiscoverable piece of content very high. A store can sell the best goods, but if they put blinds in front of the window, do not promote or advertise, make customers crawl over obstacles to place their order, then such a store will simply not do very well.

>optimizing for good search results and optimizing for high ad clicks from searchers are two different things, and you can't do both.

Why not? Major sites both optimize organic results and their SEM. AirBnb could create content for each major city they host in and rank organically. They could advertise locally or very targeted to people interested in making use of their services and link to the actual property page or listings.

>I mean parasite in the nicest possible sense and I think it's an apt description. I'm not here for the business owners - I came for the hackers.

Google compresses nearly every information in the world to a single search query. That search box is the shortest program to expand to any related content that is currently found in the world. A giant information retrieval intelligence sends out its bots to crawl information every second and store it in global time consistent databases. From a black box algorithm with supposedly 200+ ranking factors, an SEO has to give each web page the optimal chance to rank for what its worth, by reading public documentation, experiment, analyze, track, predict, format HTML documents in an information retrieval friendly way, do split testing with contextual bandits or multivariate A/B, improve accessibility, improve the link graphs of the internet and semantic web with metadata. You come for the hackers, yet you treat SEO's and Google like script kiddies.