A painful tale of SEO, spam and Google's role in all

My god, the quote from that "white hat" SEO guy just defies belief:<p>&#62; There will certainly be webmasters out there who will strip you down to the bone asking for money in exchange of link removals. These are the most soulless snake oil salesmen on earth<p>To say that about webmasters, already victims of years of abusive SEO spamming, when they then refuse to help an abusive site clean up its own mess for free .. I have no words. Could anyone possibly be more of a self-interested, myopic, egocentric prick?<p>My rock-bottom opinion of pretty much anyone who has anything to do with active SEO is re-confirmed for the thousandth time.

Blaming google for seo spam is not productive - spammers are legion and in constant battle to game google's ranking system, whatever it currently is. This won't change as long as google is used to search for things.<p>There is a solution to his particular problem: better forum software.<p>I don't want to trivialize the problem of writing spam-resistant forum software but it's not such an insurmontable problem (this forum being a proof positive for that).<p>For the reference: I've been running a fairly popular forum (<a href="http://forums.fofou.org/sumatrapdf" rel="nofollow">http://forums.fofou.org/sumatrapdf</a>) for several years, using forum software that I wrote.<p>I don't even require the user to log in, I don't require moderation for posts and yet I had zero automatic spam (I attribute this to my unusual captcha <a href="http://blog.kowalczyk.info/article/16fw/Best-captcha-is-exotic-captcha.html" rel="nofollow">http://blog.kowalczyk.info/article/16fw/Best-captcha-is-exot...</a>).<p>I occasionally get human spam i.e. someone writes a post with the only purpose of linking to some other website. I just hellban them after I see the post in my rss reader.<p>And I didn't even write any special anti-spam code (other than hell-banning), because I'm lazy. I can easily come up with simple ideas e.g. putting all posts that contain links in moderation queue.<p>To reiterate: his problems were caused by a crappy forum software that didn't do much to protect from seo spam.<p>I don't really know how current best off-the-shelf forums fare in this respect.<p>I would rather not spend my time maintaining my own forum software so I have high hopes for <a href="http://www.discourse.org" rel="nofollow">http://www.discourse.org</a>. I'm sure StackOverflow had plenty of spam problems so discourse people should understand the problem.

One way to combat this problem is to deny new users the ability to post links by replacing the link text with [removed]. HN has a karma system with thresholds that must be passed in order to get additional privileges (like down-voting). If you have a site that assigns karma to users (even if it's a secret number) then you can set a threshold for allowing links to pass through. It's not perfect but it's better than letting brand new accounts post links, IMO.<p>Another solution is to hide all links when a page is viewed by a user who isn't logged in.

I saw the SEO post he's referring to (it was one of the articles in SEOMoz's top 10 monthly email). Site owners don't realize the amount of work and headache they cause forum communities when they contract out SEO work without an understanding of what that work entails. Or they do and just don't care.<p>Google's Penguin update didn't deter the spammers, either. Here we are nearly a year later and I'm still cleaning out accounts created en masse by XRumer or other bots.

I can't put words in the mouth of the SEO quoted, but (unless you know he was one of the people making removal requests against your site) I suspect he is not talking about you or sites like yours.<p>I should add that I don't agree with the rhetoric btw, but I think he is targeting a different kind of webmaster.<p>I think he is referring to webmasters who sold links (knowingly outside the guidelines) for years. They would previously have instantly removed the links if someone stopped paying.<p>As soon as Google stepped up their game and removing those links was important, those same webmasters started charging to take down the same links.<p>I would personally point to the irony of this (google creating a market that enriches people who have been abusing their system for years) rather than calling it immoral. Ymmv.<p>Hope that helps clarify some things and I hope I'm not distorting the guy's real meaning.<p>(written on my phone. Please excuse typos).

How about taking inspiration from D&#38;B[1]?<p>Don't charge for link removal; charge for <i>priority</i> link removal. Same day service: $1,000 per link. One week: $500. Etc.<p>Free link removal: First-come, first-serve at your own leisurely pace.<p>Sell the old forum to someone else and let them handle the requests.<p>BTW, PocoMail was a godsend back in the day. Thanks.<p>[1]:<a href="http://news.ycombinator.com/item?id=4994246" rel="nofollow">http://news.ycombinator.com/item?id=4994246</a>

So this "SEO" who helping his clients to "clean up their profiles" doesn't even know how to disavow links [1]. Well, it's not surprising that somebody who hired spammers once would hire another idiot later.<p>[1] <a href="http://www.youtube.com/watch?v=393nmCYFRtA" rel="nofollow">http://www.youtube.com/watch?v=393nmCYFRtA</a> and <a href="https://www.google.com/webmasters/tools/disavow-links-main" rel="nofollow">https://www.google.com/webmasters/tools/disavow-links-main</a> (login required)

This is a great post that I'm sure rings true with many 'a webmaster. Time is money. There's no two ways about it. If you want a webmaster to modify their site in your favor, you should be willing to compensate them for their time in doing so.

You could shoot the hostage and add rel='nofollow' to all outbound links. You should be able to programmatically do this.<p>Then with an auto-reply (or 'link policy' page) inform 3rd party sites that the link-juice (good or bad) is no longer flowing.

I'd be tempted to ask for money as well, but I feel like at the very least, you should ask for an apology. They may not have known that their SEO agency was using shady tactics, but since times changed, and it's obvious that they put bad links out there, it seems like an apology would be more than a token gesture.

The problem here is the work. The solution is to write a program. Have the white hat SEO people write a program that spiders the site with admin privileges and removes offending posts. It should come with a "dry run" mode that lets you spot check it. When they get it right, you can run it for them. It's a win-win: your forums get cleaned up and they did the work.

If your business relies on SEO or is affected by Google search rank in a critical way, it may not be a viable business long-term.<p>Yes, that means I'm talking about a lot of online businesses.<p>The intelligent thing to do is sell a product or service that has value on its own and neither relies on SEO nor is it likely to be blames by other sites or companies for lowering their SEO.<p>I feel bad for these folks, but if you are planning on starting a business that doesn't really provide much value on its own that is identifyable outside of the roach motel of SEO, then you are headed into the ocean in a dingy with a small outboard motor, imo.

I've received similar emails. I've also been tempted to charge a fee for removal. I just ignore the emails.

Link removal requests can also be malicious. A blackhat seo will check his competitors domain to see if there's a spf record, if they're signing their mail, and if there's a catch-all (simply by checking if random mail is accepted). If there's neither they get a list of backlinks, from public web-crawls or sites like ahref, and request these links be taken down by sending mail with spoofed email addresses.

This is sad, sad, sad. I wonder how much awesome communities died that way. I stopped visiting Orkut when communities there got overran with spam. Also the same apply to some Usenet groups and Google Groups I used to like.

The solution is obvious -- write a routine that automatically goes through the entire forum database and disables all the links -- leave the names, but rewrite the links so they're just text, not hyperlinks. Sort of like:<p>s!<a href="http://!!g" rel="nofollow">http://!!g</a><p>The above deletes the "<a href="http://" rel="nofollow">http://</a> prefix, but leaves the original destination name, in case anyone wants to object that their post has been edited after the fact. So technically, it's no longer a link back to the originating site, but it's otherwise unchanged.<p>No human intervention required. Problem solved.

There must be a programmatic solution to this problem.<p>Do some outbound links have value on this forum? If not, then you could remove all links, or remove the "link" part of the link (change @href to text).<p>If some outbound links have value you need to identify those, and it's more complex, but a Bayesian analysis of posts should be able to score posts on their "spaminess" and remove the links on only the most spam-like comments.<p>There may be some false-positive doing this, but since no information is actually removed (only the links, not the content) it should be quite ok.

There's a difference between buying links, which Google actively started prohibiting at a certain point, and the automated forum spam you are mentioning. The latter is done using software like XRumer, and has always been penalized by Google. The former wasn't prohibited in the beginning, when link deals were often mutual agreements, most of the time involving a traffic component as well.<p>The snake oil salesmen that are mentioned, are the ones who actively participated in the scheme by selling links and making a buck, and are trying to make another quick buck now that the rules have changed.<p>Moreover, some links aren't even paid at all, but just look manipulative. For example, if you developed a wordpress theme, and your link is in the footer of tons of blogs, you might get penalized for manipulating the anchor text of your links in a non-natural way. In those legitimate situations, webmasters do have a moral obligation to cooperate.<p>I don't think anyone would think that of an honest entrepreneur being spammed to death by link spamming software.<p>On a side note, there are plenty of forums on the web that have survived the spam wave, if it were core to your business, you could have protected yourself.

Wow, so Google penalizes a site if a link to that site shows up in spammy pages? That seems like a new business model for black-hat SEOers: "hey, nice site you have there - it'd be a shame if links to it started appearing all over my spammy network - $$$ will make sure that doesn't happen." Really search engines should just give zero weight rather than negative weight to links from spammy sites.

Here is a question for those who know: is there anything forum regulars can do when we see spam posts beyond letting mods know about it?<p>I was active in a now dead forum that would get hit once or twice a week. The mods would clean it up with in a half a day but until then those posts would just sit. Does the Google web crawler consider words in replies to the spam post? I used to reply occasionally with words like: scam, fraud, got ripped off. I have always wondered if I was wasting my time. A few times I checked the link to see if it went some where legitimate (Google the base url), if so, I then searched the site for an informational web form. If there was one, left a message that their SEO company was using sleazy methods with a link to the forum post.<p>Google ought to make a code phrase that forum users can use to red flag spam posts. Though some of the posts were for Japanese and Chinese sites. The spam might not have been meant for Google but other search engines...

I had the same problem myself. In fact I still run a couple of forums. And I get at least 3 "remove my links" mails per day. I guess you can't ask them money, but they already cost you money (to maintain their spam and since the panda update read their emails asking you to remove it) so I guess the best solution is just ignore those mails.<p>Regarding anti spam I am afraid there is nothing you can do. Real humans will create an account for 0.01$ and they will post anything (I tried adding custom code in the post code as well, but those signups are not necessarily bots) In fact they even post if you don't allow them to post links (they post the urls with no http)<p>Next solution I'll try is social integration. Maybe that would work for some time but even this way spammers will find their way to create thousands of crap accounts in FB, twitter, etc.

Place a meta noindex, nofollow on the whole forum or at least the infected areas such as memberlist and user profiles. That could be done within 15 minutes and then you can answer with a standard reply that you did that to all emails you receive about removing links.

Does anyone else think that the author should set up something like mailchimp with an autoresponder to deal with the spam? Something that explains his policy, why he doesn't respond to emails, why he won't remove the link without charging them?<p>Many of the people who will have these spammy links on his forum will have paid one of the more scummy SEOs out there to raise their SERPs. The people emailing may not have originally been aware that the links have been put up. By having an autoresponder address for people to mail to this should alleviate the spam for all but the dumbest of people (that can't read what to sign up to to get a response, or those that fail to read a clearly defined policy via email).

Would love to hear feedback from the white-hat to this article.<p>Maybe get a decent discussion going from both sides. It does seem that the white^hat's remarks are very harsh, I wonder if he has honestly never thought about it from the other side's point of view.

I use XenForo and it's relatively simple to stop all forum spam for the most part. I just set up a custom captcha that only people who are interested in the topic would know the answer to, and then use XenForo's built in spam catching system for the rest...though I haven't had anything get through yet once I made the changes.<p>Also deleted forum footprint to dodge people scraping my forum off of Google, so that takes care of about 99% of all spam attempts. Someone would have to custom register accounts on my forum to get in, after which they would be destroyed by the spam catcher :)

Maybe we should not blame search engines, but SEO really has bad effects on the Internet...<p>PS, due to spams I've closed this forum (<a href="http://writingoutliner.com/forum" rel="nofollow">http://writingoutliner.com/forum</a>), a online forum for my 'Addon documents organizer for MS Word'.<p>And I regret I use BBPress as the forum software, because new 2.x versions are not compatible with the old 1.x versions...

Maybe it could be scripted? Site owners would have to place some proof of ownership on their site (to be generated on the site of the forum owner - see Google web admins site verification), then a script could detect the faulty links on the forum and remove them automatically.

Oh man that is some sweet irony.<p>Just wait till the next waive of spam from SEOs trying to get competitors penalized then the competitors firing back and contacting you. Google stirred up a hornets nest because they couldn't figure out how to effectively devalue these links completely.

There exist very popular forums that receive no such spam.<p>Hacker News comments also seem free from it.<p>Any idea why it is that some forums get targeted with 10000's of spam accounts, while others don't? What is the trick to protecting if even capchas don't work?

Why are people getting bent out of shape over some bad links? Can't you just disavow them in Google Webmaster Tools?