How One Check Box Lowered Conversions by 17%

I think that for a lot of A/B testing software (if not all) the statistics is not done correctly.<p>Here is what people do: They keep running the test till they feel that results are statistically meaningful. That is that they do not fix the N apriori.<p>But that is wrong.<p>Put another way if you wanted to test if a coin was biased you should not stop the test after 10 tosses if you got 9 heads and say the coin is biased. you should decide before hand how many tosses (N) and then conduct the test.<p>And if results are inconclusive you need to flush the data and start again

In the screenshot, the checkbox is actually checked off. Was that the default setting? If so, I could see that being off-putting to users -- whenever I see that, I assume they're trying to trick me into opting in for something.

The sentence <i>After only a short period of time we were able to determine with 94% confidence that the variant was performing worse...</i> sounds like they might have fallen into the trap described here: <a href="http://www.evanmiller.org/how-not-to-run-an-ab-test.html" rel="nofollow">http://www.evanmiller.org/how-not-to-run-an-ab-test.html</a>

I'm surprised no one has mentioned this but the "x" isn't very confidence inspiring, especially next to financial fields. Patio11 once told a tale [1] of an Indian designer who created a "continue" graphic that was red with a yellow halt-don't-continue-past-here hand. I would not be surprised if a green checkmark instead of a black "x" would have helped conversions, but this is coming from someone who has not done any A-B tests...<p>[1] <a href="http://www.kalzumeus.com/2008/03/02/my-experience-with-outsourcing-web-design/" rel="nofollow">http://www.kalzumeus.com/2008/03/02/my-experience-with-outso...</a>

The small things really do matter when dealing with fickle users. I wonder, though, how much of the 17% might be due to the placement of the checkbox. Putting it below the credit card info would make me think "wait, I'm paying these guys and they are telling me it isn't easy to use?". Perhaps moving it up by the email address field would yield different results.

I would suspect that users are responding to your headline message "Get 5 tip emails on how to use Skritter" with concern that your product has a steep learning curve - I have to read 5 whole emails to figure out how to use the product? Yug.<p>The difference between a good headline for opt-in vs. a bad headline will account for decreases like you experienced, but a good one can actually increase your conversion rate.<p>Trying something less offensive, or even pushing in an active voice would be a better fit:<p>[X] Get free tips and bonuses from the Skritter newsletter.<p>or<p>[X] YES - I'd like the most recent news &#38; updates on Skritter delivered to my inbox.<p>The following article about some of the resorts we drove is not perfectly relevant, but it demonstrates how the addition of a checkbox bumped results up by 11%:<p><a href="http://conversionvoodoo.com/blog/2010/07/11-conversion-rate-increase-with-a-%E2%80%9Ccommitment-checkbox%E2%80%9D/" rel="nofollow">http://conversionvoodoo.com/blog/2010/07/11-conversion-rate-...</a>

"Get 5 tip emails on how to use Skritter"<p>is very hard to place a value on, to a customer who hasn't even <i>used</i> Skritter yet (they are still at the registration stage!).<p>why not<p>"Get our <i>Secret Tips to Language Learning</i> free course (5 emails)"<p>or something like that. the benefit sounds much more general, although the content can probably be basically the same as it is now.

94% is not science. If you do 15 random experiments, there's a good chance one of them will succeed at the 94% confidence level.

Expedia had a similar problem (<a href="http://www.silicon.com/management/sales-and-marketing/2010/11/01/expedia-on-how-one-extra-data-field-can-cost-12m-39746554/" rel="nofollow">http://www.silicon.com/management/sales-and-marketing/2010/1...</a>) with a different datafield, but the message is clear - use thy analytics well :-)

I don't think it's the presence of the checkbox so much as the reminder that "we will now have your email address and you will receive generic mail from us". I don't think I've ever heard of anyone /excited/ at the prospect of receiving generic mail from vendors, and certainly know of plenty who hate it.