Rap Genius is Back on Google

Honestly, this is just saddening. Google needs to have a consistent policy towards these blackhat SEO offenders and enforce it for everyone. RapGenius bought backlinks to artificially inflate their pagerank. They knew they were cheating but they did it anyway. They got caught. And they were hit with a manual action on Google. Sounds good so far. But that&#x27;s where it stops. RapGenius, using the connections they have from their multi-million dollars in VC funding, got back on Google and was able to avoid paying the price for their cheating.<p>Compare this to regular websites. Lots of smaller sites will end up paying a &quot;whitehat SEO&quot; firm to work on a &quot;link strategy&quot;. These firms will claim up down and sideways that what they do is legal, ethical and follows Google&#x27;s rules. But, what they actually do is either (1) create networks of fake sites to provide backlinks on certain terms to artificially boost the site, (2) place spam comments using bots on legitimate sites to do the same [not that this will thankfully no longer work well due to the latest Google algorithm update], or (3) pay legit sites to place backlinks to artificially transfer pagerank the same way that RapGenius did. Now, these other sites, when they get caught, they get a manual action or a smackdown. The difference? They have to actually pay the penalty. Arguing that they didn&#x27;t know usually doesn&#x27;t work. The penalty is LONG. They can&#x27;t call on their VC firm to make calls at Google to give them a get out of jail free card.<p>I&#x27;d like to call on Google to create a public policy on how they handle these manual actions with some clearly defined penalties (example: 3 month manual action of 6 PR drop, etc) and to consistently enforce them across the board. That way a mom and pop site that pays a &#x27;whitehat SEO firm&#x27; and gets caught doesn&#x27;t have a worse time than a site like RapGenius that purposely engages in blackhat SEO who can use their VC connections to get out of having to pay a penalty in under 2 weeks. It&#x27;s also sad that Google gave this get out of jail free card to a site whose entire business model is based around other people&#x27;s copyrighted works which RapGenius doesn&#x27;t have a license for, doesn&#x27;t pay for, and publishes illegally.

This is great for Rap Genius.<p>But this truly saddens me as well. I cannot for the life of me understand how they were able to fix ten-of-thousands of backlinks and other shady practices and get back in the good graces of Google so darn quickly.<p>Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail.<p>Could it be that Rap Genius was special because it has Andreessen Horowitz?<p>Sure seems that way.<p>I am not going to stop using Google, but I am inclined to use them lesser now.<p>I would like to hear why, how rap genius was fixed this and what this means for other people who get penalized from Matt Cutts though!!

Complete and utter bullshit.<p>That they blatantly employed blackhat SEO techniques and are back on Google within a matter of days is sickening.<p>Think of all the sites that are penalized in various ways for similarly shady SEO practices and have to spend _months_ to regain their previous position.<p>Hopefully Bing et al can capture some marketshare and bring some degree of neutrality to the search engine landscape (i.e. not have every website owner on the planet beholden to the beast).

I do not understand why people are offended that they came back quickly AFTER doing a bit of penance.<p>Should their investors not have made calls? However, it should be noted that they did their part.<p>One of my favorite quotes is this: &quot;To save a drowning man, he must first give you his hand&quot;<p>If the Rap Genius guys were incompetent and did not do their part in removing the links so quickly (see main story for technical details) there would be little or nothing their investors would have been able to do about that.<p>BTW, it also helped that people really found Rap Genius useful. I&#x27;m sure many searched had rapgenius appended to it. So the lesson is, make your app so good that when Google delists you, it will make them look bad.<p>Thankfully?, there is also a precedence for other offenders to use. if you can detail you have atoned for your sins and Google insists your you must do a certain time, you know where to turn to.<p>BTW, this is Hackernews, and I look forward to the discussion of the (de)merits of how they scrapped and analyzed 177,000 links and not espousing of anger that they survived the punishment.<p>I&#x27;m sure Mahbod (my favorite) and co have learned that you do not go &#x27;daaawging&#x27; when you are in a hole. I wish them success.

God this pisses me off to no end. So my site is targeted by negative SEO. (Yeah, I know, I know, negative SEO doesn&#x27;t exist. Fuck off.) So who do I call? I&#x27;ve been penalized for months and there is nothing I can do about it. Nobody can help. But these fuckers are able to make a call and recover from a SELF INFLICTED wound in less than a week. I am being targeted by crap that is COMPLETELY OUT OF MY CONTROL and it&#x27;s tough luck little buddy. The double standard here is fucking sickening.

This post is complete bullshit. Of course they have relations. Maybe they cleaned their backlinks, but still I cannot believe how they got back so fast in SERP.<p>Anyway, Franz Kafka would die again if he saw his text on this site (<a href="http://poetry.rapgenius.com/Franz-kafka-a-dream-lyrics" rel="nofollow">http:&#x2F;&#x2F;poetry.rapgenius.com&#x2F;Franz-kafka-a-dream-lyrics</a>), so let&#x27;s just forget about this site and don&#x27;t use it.

One thing I know is that RapGenius is good about making a big stink, being in the news, and then having things work out in their favor, even if they were the ones doing a dumb thing in the first place.<p>They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku?<p>They did some incredibly dumb SEO things because they are greedy and are willing to cheat to win. That is on RapGenius, but once they got banned it is somehow Google&#x27;s fault and they should be reincluded? I don&#x27;t get it.<p>RapGenius is great at one thing - creating controversy and getting press for it. I guess that works for them, but it seems like a very selfish way to grow a business. Instead of focusing on building value themselves, they are willing to tear down their business partners in public to get what they want.<p>I&#x27;m not sure why this behavior hasn&#x27;t put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.

They have the word &#x27;lyrics&#x27; at the end of the slug for that post and a number of others which don&#x27;t have any relevance to lyrics. Surely that&#x27;s still bad SEO practice?

Its interesting that despite their founders outspokeness, the influx of heavy weight VC, tech press expose&#x27;s on their offices, etc., RapGenius is really no more or no less than a standard, multi-multi-page SEO site, no different than the countless others just like it. No brand, no user affinity, no direct traffic to speak of. Just a bunch of drift net pages crawled by Google waiting to pick off unsuspecting search traffic. My favorite line of this post was the last: &quot;Much love. iOS app dropping next week!&quot; - unless you are searching that term in google, Im not sure who that is addressed to.

I wonder how long it would take for any other website without 15 mln backing and direct line to Google to return.

Pretty interesting decision to include a lengthy &quot;technical digression&quot; in their SEO apology:<p>&lt;&lt; Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. &gt;&gt;

So which hit should google show for a query like &quot;the fox lyrics&quot;? Of course, it should be a page that contains the lyrics, but which one? The most popular one? The one shown by google will be the most popular one.<p>From an IR perspective, search for lyrics is somewhat strange. I don&#x27;t think it&#x27;s easy to say if some hit is more relevant than the other. Sure, more popular sites may offer better UX (or simply less annoying ads), but at the end of the day it&#x27;s a search for facts.<p>I wouldn&#x27;t be surprised if google will serve those lyrics themselves some day. Freebase (and hence the google knowledge graph) already contains tons and tons of information on pratically any popular song (organized in a confusion way via recordings, canonical versions and whatnot) - artist, genre, album, release dates, length of a recording, etc. I don&#x27;t think adding the lyrics as well is totally out of question. After all multible KBs are user curated and users provides lyrics.<p>Just like they serve &quot;fact-hits&quot; from their knowledge graph for queries like &quot;movies by spielberg&quot;, google should then be able to serve the lyrics directly. While this is bad for lyrics sites, imho it is convenient for users and follows the same prinicple like &quot;movies by spielberg&quot; or &quot;wife tom cruise&quot;

People should worry why Google has so much power nowadays, rather than why they unban Rap Genius so fast.

This really shows that even in Silicon Valley, your success is driven as much by the connections you have and the people you know, rather than pure talent and merit.

The &quot;technical digression&quot; with Github code on how to do a mass scrape is cool, though not surprising (er...how else would you accomplish such a big data task...though I&#x27;m glad I&#x27;m not the only one who uses Typhoeus, a great but underused gem).<p>But what&#x27;s surprising to me is that a company that knows how to automate things and work with data...<i>why the f--- were they manually soliciting bloggers to post links?</i> That seems seriously inefficient and comically backwards...as if Sergey Brin spent his early Google days modifying SERPs by editing and uploading Excel spreadsheets to the server.<p>It seems seriously inefficient to me because at this point, Rap Genius has a lot of traffic and a lot of linkage already. So did the few backlinks, per day, that RapGenius got, through lame blogmasters who would participate in such a thing...really have an effect on their SEO rank, i.e. is it so easy to still game Google?<p>Or, was RG just doing it because they <i>thought</i> it would have a worthwhile effect (keep in mind that this is one of their co-founders who was individually emailing people), i.e. RG was being naively optimistic?

Looks like this involved quite a bit of work on Rapgenius&#x27; part but I am glad they cleaned up their mess and apologized to Google.<p>Moreover, this article feels pretty neutral&#x2F;humble and techy, a far cry from the haughty attitude their founders usually display in interviews.

<a href="http://m.entrepreneur.com/article/230631" rel="nofollow">http:&#x2F;&#x2F;m.entrepreneur.com&#x2F;article&#x2F;230631</a><p>But now, Rap Genius co-founder Ilan Zechory tells us the site has returned to Google&#x27;s search results. &quot;It takes a couple days for Google to re-index everything, so search results are a little wonky right now, but we are officially reinstated,&quot; Zechory said in an email.<p>Somewhat ironically, Zechory served as a project manager at Google for two years prior to founding Rap Genius in 2009.<p>&gt;&gt; Wow that is troubling. So site that gets banned gets unbanned quickly and founder just happens to be ex google project manager

I&#x27;ve followed SEO on and off for years and I wouldn&#x27;t have thought that their strategy of allowing people to generate and embed links back to the site was shady at all. I even recommended something similar a few years ago in good faith.<p>They&#x27;re useful, relevant, and it just seems like good marketing. These things will spread organically and drive traffic even if you remove the SEO benefits.<p>Could you say that Scribd or Slideshare are doing something similar by allowing people to embed documents in their websites along with a no follow link back to their site?

Clear as day there is a double standard. That&#x27;s what should be discussed.

Actually, it&#x27;s understandable that Google set this on High Priority...<p>#1: it&#x27;s a high traffic website, that is a startup and the whole deal got a lot of attention of us (HN Readers) + Matt even saw this here. So it was good to respond fast (no choice)<p>#2: RapGenius is dependant on Google (although they try to let you see it as a social network, but it&#x27;s not), they would have gone bankrupt ( i suppose) without Google.<p>#3: Without a quick response and&#x2F;or measure to fix their shit on their end, investors wouldn&#x27;t invest any more money in RapGenius and Google would probably hurt a lot of future investments (how is this startup dependent on Google, ...)<p>BMW once got a penalty from Google, they didn&#x27;t got easily off. But their business wasn&#x27;t dependent on Google.<p>Either way, there are both pro and cons to each action. But overall, i think they handled if fairly well.

I am sure they went through many sleepless nights during the holidays. Hella sucks. I think Google is doing the right thing and I am sure not just related to seo that rap genius learned a lot about themselves and will become stronger... as well will the overall tech community.

Disavow tool actions take months to bring any effect(if any).

Interesting closing paragraph:<p>&gt;<i>Though Google is an extremely important part of helping people discover and navigate Rap Genius, we hope that this ordeal will make fans see that Rap Genius is more than a Google-access-only website. The only way to fully appreciate and benefit from Rap Genius is to sign up for an account and use Rap Genius – not as a substitute for Wikipedia or lyrics sites, but as a social network where curious and intelligent people gather to socialize and engage in close reading of text.</i><p>ie, you should really just become a member rather than relying on google...<p>Also, where was bing in all of this?!? ;)

what does not make sense to me is that after removing all the back-linked urls, should not that drop Rap Genius&#x27;s overall ranking even lower than before, since the artificial boosting has been removed. Why they are back on the top. googling &quot;lyrics&quot;, RP is number #6 in the first page. I think RP has received special treatment here. Google search team needs to clarify this.

Well atleast now AH partners can get back to posting rap lyrics on twitter. We wouldn&#x27;t want anything to stop that bit of entertainment.

The tech part is obvsiouly here to calm down people from Hacker News and make them say &quot;Oh this part was good, after all they are not so bad&quot;. It&#x27;s just a shitty scraper but they talk about it like it was a revolutionnary script...<p>There is no way they could get back so fast and so easily on google.

The SEO strats &#x2F; growth hacks weren&#x27;t entirely misguided. Rap Genius just didn&#x27;t perfect their activity and didn&#x27;t let it exist entirely behind closed doors. If Rap Genius wanted to risk it and engage in blackhat-like activity, it should have at least kept things more quiet and less spammy. Rap Genius should have kept the finer details, at least, to a phone call with the blogger.<p>Google can&#x27;t see a local business handing their customers&#x2F;client a $25 gift certificate in-person for a random positive Google+ review. Along the same thread, Google would also not be able to see Rap Genius flying out a community outreach rep to meet with a prolific music blogger at a bar to go over an exchange for a positive mention and authoritative link. Just saying.

Were the 178k URLs that they scraped on 178k unique domains or a very few domains?<p>The tool they wrote looks like it would have been a DoS attack if the latter were the case. There appears to be no thought or consideration for the sites they scraped that had linked to them, no attempt to read robots.txt and see whether they should fetch the pages.<p>What they did was wrong in the first place, but they could try and do things better in their attempt to repair things. Instead they continue to act in a way that externalises any cost to them and continues to treat their service as somehow privileged enough to deserve such breaks.<p>How many of the 53k failures to pull and scrape pages were to do with the fact that the servers in question were under a DoS attack by RapGenius?

Probably the story is missing the help of a highly skilled SEO consultant&#x2F;company but it&#x27;s impressive anyway. To get it done so quickly they had to work 24&#x2F;7. Kudos!

I have to note, you have to be big to make comeback like this. Google provided disavow tool, but from my experience it simply not working. So it is manual action for sure.

Infinite scroll (or some type of weird JS) won&#x27;t let me get to the bottom of the page... Ain&#x27;t got ish to do with this but I just thought that I should mention.

If Google were to have a year long ban on RG stuff to leave a precedent, Would people start to view Google as a sub-par search engine because Bing returns &quot;high quality&quot; results like RG still?<p>It&#x27;s also rather weird how a lot of people seem to view Google as this public service for the people by the people etc, it&#x27;s not. They can do whatever the hell they want and if you don&#x27;t like it, simply stop using their service to send a message.

I think it will take time for them to regain their rankings for non-branded keywords.<p>I just searched for justin Bieber lyrics and they are sitting in middle of page 3 on google.<p>They are ranking for branded searches, but not money keywords.<p>Although long term, I see this being a net positive for them due to the thousands of backlinks they got from all the press around this story.<p>I see it taking them several months to regain rankings...especially if the links they disavowed were artificially propping them up.

While we&#x27;re on this topic, are GroupOn&#x27;s API Branding Requirements in violation of Google&#x27;s webmaster policies?<p><a href="http://www.groupon.com/pages/api-branding-requirements" rel="nofollow">http:&#x2F;&#x2F;www.groupon.com&#x2F;pages&#x2F;api-branding-requirements</a><p>They make it a requirement of using the API that &quot;nofollow&quot; is not used: &quot;Don’t ‘nofollow’ your links to Groupon.&quot;

Is it a sign of getting older when black background web sites get harder and harder to look at for longer periods of time?

I&#x27;m waiting for their next SEO single. &quot;Oops, I did it again!&quot;

Snapchat could learn a thing or two of how to do PR from this post.

Rap Genius appears to have offset their drop in search results traffic with all the publicity they&#x27;ve been getting from being penalized by Google.

I give them a lot of credit for being transparent with their learning mistake, taking ownership, and describing what they did to fix it.

&quot;tis better to apologize later, than ask for permission first.&quot;

This is a relief. RG certainly gives the best search results for a lot of queries.<p>This whole story made me fear that Google was choosing power over user experience.<p>And that should never be the case, no matter how offending the founders of RG might appear sometimes. Don&#x27;t be evil!

Money wins every time.

Certainly a double standard exists at Google. I originally thought finally Google is taking a firm stance, but this reversal only does harm to Google&#x27;s image imho. Take home message is, blackhat SEO is forgiven if you are funded by connected people. Don&#x27;t have funding or connections but make the slightest infraction in SEO? Feel the google girth.