How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

I'm wondering if merely displaying a number is really "social proof" in the Cialdinian sense.<p>Possibly at one time, but now I think the "proof" part of social proof must be demonstrably not BS for it to work. In the same way that testimonials by "J.S., Ohio" have no real meaning anymore, but "Jared Sanderson, CEO of Whatever.com" would probably be taken seriously.<p>An interesting experiment would be to get permission from some of your better-known users to say something like "Join Bill Gates, Dalai Lama, and 14,152 others...". Possibly even better to use little avatars or pics of a bunch of people, ala Facebook Like or the old BlogLog widget. Who knows? I feel like that'd be a better thumbs up/down on social proof than just the number.

I wish I knew more about statistics. Punching just the two variants that made it to the end into a Chi square calculator gave me a 90% confidence, not the 95% he cited.<p>I also wonder about stopping the variation early, and even the conditions under which he stopped the test. <a href="http://www.evanmiller.org/how-not-to-run-an-ab-test.html" rel="nofollow">http://www.evanmiller.org/how-not-to-run-an-ab-test.html</a> talks about pitfalls when "testing until you reach confidence".<p>Are there any better references for these types of questions?

Not sure if it's significant yet. What he really got was: 9 people click on the default form, 5 people on variation one, and 14 people on variation two. These are very low numbers no matter what way you look at them.

Not only is it a distraction. I see that and think "You are going to send me email that will be sent to 15,000 other people? That would just be spam for me! Why should I sign up if you are going to spam me?"

The devil's in the details, but the angels are there, too.<p>Amazing how much impact these "minor" changes can have on the bottom line.<p>Thanks for the reminder to split test all things great and small.

I found it very amusing that after the long post demonstrating that the "social proof" line was cutting his sign-ups in half, there was a box asking me to sign up which included the social proof.

It looks to me as though they should try the original headline without the social proof. Since the Control beat Variation 1, but Variation 2 beat both, that would seem to be a logical next step.