How to stop comment spam [bjorkoy.com]

How to stop comment spam

Posted on January 06, 2008. 39 comments.

Comment spam must be the number one threat against an enjoyable blogging experience. Many people quit blogging entirely because they get tired of fighting the ever evolving spam bots. There are, however, ways to win this arms race.

Below I’ve outlined the techniques I use to keep this blog spam free. I’d be very surprised if I ever get spam from bots again - this stack of techniques should be quite bulletproof. The methods are listed in order of usage.

If a comment passes all these steps without being flagged as spam, it’s allowed into the database.

It should be noted that I have no patience for manually approving comments - I feel like I’m letting the spammers win. For shame! :)

Step 1: “Do” field

(This step was initially about referers, but as pointed out in the comments, all browsers do not send referers, making this a bad idea. Here’s a different method.)

We’ll first check if the HTTP POST actually comes from the blog entry in question. This is easily done with a hidden form field called “do”, which contains a simple value of your choice.

When a comment gets posted, you check if the field “do” is set and equals the chosen value. As spammers often send POSTs directly to your “add comment” URL, this might stop some of the less ambitious culprits. The HTML could look something like this:

<input type="hidden" name="do" value="some_random_rotated_value">

The spam check (in Rails):

if params[:do] != the_random_rotated_value
  spam = true
end

It won’t stop all spam, but the philosophy here is that if it can stop any spam, we’ll do it. Onwards, to level two!

Step 2: Hidden Turing test

A recent trend amongst blogs is to have a simple question as part of the comment posting experience. A simple CAPTCHA, in other words. The user has to answer a question, and the answer is then used to decide if the poster is human. In effect, the Turing test expressed through a single form field.

The problem is that this diminishes the user experience. Any extra field a user has to fill in order to leave a comment is just another reason for that user not to comment at all. And since you want as many comments as you can from as happy users as possible, we should do something about this.

Here’s my take: You use a question field, but with an added twist. Through CSS, hide that field. In your code, check the posted value of this field. If it’s not empty and does not equal the answer to your test question, the comment is spam.

You’ll also want to give the field a name that makes it certain that a spam bot will fill it out. I use “url” to store links, so here I’ve used the name “website”. The HTML looks like this:

<div style="visibility:hidden; display:none;">
  <label>What is 2+3?
  <input type="text" name="website"></label>
</div>

(Jan Pingel and Jeremy Weathers suggests in the comments that visibility:hidden is needed in addition to display:none in this case, as it will help apps like screen readers to hide the test.)

And the POST check might look like this (at least in Rails):

if params[:website] != '' and params[:website] != '5'
  spam = true
end

We hide the field because common sense says that spam bots probably won’t parse CSS, and therefore fill in the field based on the name given. A user without CSS turned on will see the field, and fill inn the answer to your question. The vast majority of users, however, will never even see the question, thus making the probability of them leaving a comment higher.

A truly resilient spam bot might pass this test. It might parse the CSS or understand the question. No matter, on to level three!

Step 3: Akismet

Akismet is a great tool for stopping spam. It’s basically a web server to which you pass the content of and information about each comment, and it in turn gives you a “true” or “false” signifying if it thinks the comment is spam. It has a great hit rate, and has worked for a vast herd of now spam free Wordpress bloggers.

The service is open to everyone, though you’ll need to create a (free) account on wordpress.com. You’ll then get an API key which will give you access to the Akismet servers.

There’s already been written plugins and API wrappers for many systems and languages. Here’s a few:

Wordpress plugin (official)
Akismet API wrapper for Ruby
Ruby on Rails plugin (uses the above wrapper)
Drupal module
Movable Type plugin

(A simple google search for Akismet and your blogging system will likely produce your desired result.)

In many cases, using Akismet will protect your blog from almost all spam. However, since this is supposed to be a bulletproof setup, I would still recommend using all the techniques mentioned above. Since Akismet is starting to get very popular, spammers might work extra hard to find countermeasures for it, and as it’s a centralized service, it’ll probably experience some downtime.

Still not satisfied? I have one more technique you can try out.

Update: Carl Mercier writes in to mention Defensio, which is a service similar to Akismet. I haven’t tried it, but judging from it’s website, Defensio might very well be a worthy competitor. You could of course use both, which would be especially useful in case one of them is experiencing downtime.

Step 4: Click test (Javascript required)

This is a technique used by SimpleLog, and requires that Javascript is turned on by the user posting a comment.

I don’t use this, as I think websites should always work perfectly without JS. However, if you disagree, or run a site where one of the requirements is that JS must be turned on, this might be just what you’re looking for.

First, create a bunch of hidden fields, one for every required field and the submit button, like this:

<input type="hidden" name="check-1" value="no">
<input type="hidden" name="check-2" value="no">
...
<input type="hidden" name="check-n" value="no">

In every required field, create a virtual form value through Javascript, that is only set if the user actually physically (or is that virtually?) entered something in the field, like this:

<input type="text" name="email" 
onkeypress="this.form.elements['check-1'].value = 'yes'">

Do this for every required field, just change the value “check-1” between each new field, so that you get a separate check for each. You should also check if the submit button was really “clicked” by the user, like this:

<input class="submit" 
onclick="this.form.elements['check-2'].value = 'yes'>

You can then check in your “add comment” method if every one of these fields are set to the correct value (“yes”). If they’re not, the comment is spam. (Or the user has JS turned off, which is the problem I mentioned above.)

To really annoy spam bots, obfuscate the custom values, like this.form.elements['ed8mkl32'].value = '9r3w'. You can then randomize these and place them at different intervals and in different fields, all while checking for the current correct setup in your script.. Am I going too far? :)

Step 5: Random field names

Andrew suggest another simple yet effective method:

The technique I am using, and which is working very well, is to randomise the names of the form fields.

When the form is loaded a PHP script generates random names for all the form fields and then adds a hidden element with instructions on which random form name should equal which real form name.

When the form is submitted the comment handler unscrambles the names and assigns the values. Any form fields submitted that were not included in the unscramble instructions are wiped.

Spammers, be gone!

In addition to the techniques mentioned above (except step 4 and 5), I also use a blacklist that comes with SimpleLog as a last measure. I like to think that I’ve caused at least some frustration in the mind of a spammer at this point. :)

I’ve yet to get automated spam after implementing these methods, although I hope no enterprising spammer sees this post as a personal challenge, with its provocative title (in spammer circles) and all.

If anyone has any techniques to add to this list, enlighten me through the hopefully spam free comments section below.

Tags: web

39 comments

Adam Fortuna says:

One additional one I’ve seen, although it does require JS to be enabled, is to change a form variable on mouse movement. Not necessarily movement over a field or anything, but in general.

This has it’s problems when it comes to accessibility though, so might not be a 100% solution, but 90%+ at least.
Ville Säävuori says:

I’ve used hidden touring test (#2) on many sites with great success. Turns out that spam bots are very eager to fill out every detail on the comment forms, so most of the time they will fail this miserably. Together with Akismet, no worries about spam.
Jan Pingel says:

In regards to Step 2: The Hidden Touring Test – Sometimes screen readers ignore display: none; and announce the contents to the user. Adding visibility:hidden; would allow users of screen readers to comment without being pestered by the CAPTCHA.
Andrew says:

The technique I am using, and which is working very well, is to randomise the names of the form fields.

When the form is loaded a PHP script generates random names for all the form fields and then adds a hidden element with instructions on which random form name should equal which real form name.

When the form is submitted the comment handler unscrambles the names and assigns the values. Any form fields submitted that were not included in the unscramble instructions are wiped.
Olav says:

@Adam: Yeah that would work, and has about the same drawbacks as the JS method I mentioned.

@Ville: Sounds good, let’s hope it works for me as well. :)

@Jan: Thanks, I did not know that. Do you have any links/articles which explains this? Either way, I’ll update the article.

@Andrew: That’s a great method, I’ll add it to the article.
Tilman says:

Thanks for this post, this is the best list of comment spam countermeasures I’ve seen so far. Although all the solutions together is still not “bulletproof” in my book, but that’s ok! I think I will start with the hidden turing test, that’s a brilliant idea.
Peter Harkins says:

I know my browser doesn’t send referers and some proxies strip them. It’s actually fairly common. This post tests if your comment form is OK with that.
Jeremy Weathers says:

I would use both methods of hiding: “display: none; visibility: hidden;” - at least with some browsers/doctypes using only “visibility: hidden” results in an empty spot in the page where the hidden element still has allocated space.
Olav says:

@Tilman: You’re of course right. All we can hope is to get as close to stopping all automated spam as we can. Poor wording on my part.

@Peter: I wasn’t aware of this, but I actually check for something a bit different on my site, so your comment might still have gotten through. I actually found another problem with this step as well, and will update the article. Thanks. :)

@Jeremy: Good tip, I’ll add it. Thanks.
Jan Pingel says:

@Olav: Here is an article about it: Screen readers sometimes ignore display:none
Olav says:

Thanks! I should have known that such an article of course had to come from 456 Berea Street. ;)
Carl Mercier says:

May I also recommend our spam filter, Defensio? We support many blogging and development platforms. We launched just a few months ago and people like it a lot so far!
Olav says:

Thanks, Carl, Defensio looks great. I’ve updated the article. What’s your thoughts on using Defensio together with Akismet? This would probably eliminate almost all possible downtime.
Carl Mercier says:

We highly recommend that you use Defensio as your stand alone spam filtering solution.

Other spam plug-ins will only serve to lessen the opportunity that Defensio would otherwise have to get to understand your blog’s spam traffic.

(that’s the “official” response taken from our FAQ ;-) )
Pambuk says:

hidden turing test This is great, I was looking for something less nasty than captcha and I think I found it. Simple solutions are what I like best.
Piter says:

Your blog is of definite quality.
jacek says:
I was have much trouble with spammers on one of my site (guestbook) I stop all spam doing that test:

who you are:
1. input, radio, checked, value=bot
2. input, radio, NOTchecked, value=human
Who want to add post must check second input, its stop spam in 100% :D
Ellen says:

Re: Step 2: The Hidden Touring Test

Would you please tell me why I should use an element hidden with css, rather than simply use ?

Thanx
Frank Bruno says:

I stopped all my comment spam for my Wordpress blog by using “Comment Bot Nuker”. All the other plugins I tried for Wordpress only holds the comment spam from moderation. I tried them all even the “add math” and Askimet. Most all the comment spam you will get is actually from the bots that scour the Internet looking for security breaches. Using Comment Bot Nuker it nukes the comments before it is even held for moderation and lets the good comments through. So there is nothing to moderate. I don’t have to go through hundreds of comments held for moderation anymore. I foudn it here: http://videomarketingtactics.com/vmt/stop-wordpress-spam
John says:

open your forms on a popup or new window requiring a password. enter the password in the password challenge so when the aplet pops up for password it says ” enter the password ‘gofish’ to continue” this will require a small extra step to get to the form. You can vary easily change the form this way. you can see an example of this at http://painbrain.com/linkpost/authorpost.html

the code is vary simple: There are some intentional missing
John says:

open your forms on a popup or new window requiring a password. enter the password in the password challenge so when the aplet pops up for password it says ” enter the password ‘gofish’ to continue” this will require a small extra step to get to the form. You can vary easily change the form this way. you can see an example of this at http://painbrain.com/linkpost/authorpost.html

the code is vary simple: There are some intentional missing
John says:

open your forms on a popup or new window requiring a password. enter the password in the password challenge so when the aplet pops up for password it says ” enter the password ‘gofish’ to continue” this will require a small extra step to get to the form. You can vary easily change the form this way. you can see an example of this at http://painbrain.com/linkpost/authorpost.html

the code is vary simple: and you can see it by looking at page code. its the little java password page protector. This does cause some headaches for IE7 but nobody likes Microsoft anyway.
t says:

s
yo says:

if i was the president ill arrest those asshole spammers and sentence them to 5 years solitary confinement ftw!
James says:

Hi, Well at my website too i am having spam problems!! I get LINKS of nude website..lol in my inbox through the contact form and rest all on my guest book. I am slowly implementing different methods for it. Recently reduced the number of characters for textarea in contact form. Well soon it add captcha. I want to block any comments which has link on them! Only thing I am facing with is how do I recognize a link successfully. After that is true i can block it simple javascript to not submit the form. Simple huh? Well if you have a solution give me a buzz in my website. I will b so grateful :D
shepetivka says:

I agree, big mistake. I also had trouble over the holidays finding people’s gift lists online. Before finding the link was clear and easy, but i spent alot of time just trying to find where to lookup my friend’s list. Another usability mistake.
Edison says:

Thx.. I now at least got some idea how to stop the spam on my blog
Test says:

document.location.href = ‘http://www.yahoo.com/’;
Test says:

document.location.href = ‘http://www.yahoo.com/’;
Test says:

nice work just testing security
tips on stopping smoking says:

When dealing with comment spam it is very hard for me to get around it. This plugin seems like it could help me in no longer dealing with this problem which brings me to have so much spam on my blog. Is it just me or does everyone feel that the more links out you to get to a webblog the more comment spam you get. :(
Michael says:

I too am having problems with spam and would like to use step 2 (Hidden touring test) in my comments page. I copied the html code above into my comments page. Whats the post check mean and how do I add it into my comments html. Do I copie and past this code into my page to? Thanks!
betabug says:

Your list is very good. I’m doing some of the same things, will think about implementing some of the others (the “hidden turing test”). “In front” of all these blocks, I’m using the http:BL blacklist from http://projecthoneypot.org - this will stop a lot of bots especially those coming from botnets. The nice part is that they get a 403 response before I even have to process their input.

Just as you described, the “defense in depth” is set up to have the measures that are cheap for my server first, the “expensive” and “slow” stuff last. Akismet is obviously at the last step.
Michael says:

In step 2: Hidden Touring Test do i add the post check code into my email filter program on my host site? Can someone explain the what the post check code is and where I add it to my site? Sorry computer ill here lol.
Zac Davis says:

I definitely would never use the JS method.
dıs cephe says:

Thx.. I now at least got some idea how to stop the spam on my blog
hekimboard says:

I agree, big mistake. I also had trouble over the holidays finding people’s gift lists online.
jos says:

Spam is a serious issue, however be wary not to confuse legitimate comments with spam, i know some blogs that delete your comment even if you just put a link to your website.
wrmhxosogs says:

Hello my friend, your site is very good! http://zavisvsndn.com