One of my friends, who is regrettably still in stealth mode or I would be shillin' like a villain, is working on an automated AB testing framework. Suppose you had a website with a banner, a sidebar, and body text with 3 paragraphs, each with 4 possible variations.<p>His software, as far as I understand it, genetically engineers the optimal web page for conversions for you. It apparently is already running on his (unrelated) software site and he credits it with helping him push through the $1 million mark. He also says that it doesn't assume there is a static optimum, but rather that different visitors might need to see different things (e.g. hypothetically, if someone has seen your screen shot page they probably don't need more visuals to make the purchase decision, so you'd use more screen real estate on textual benefits and less on screenshots).<p>I'm going to be beta testing it on my site, and I'll be blogging progress as usual, if anyone is interested in hearing about it. I'm pretty excited. I'm told the tracking code will be ready in about a week or two and the functional bits ready for beta relatively soon after that. (Nice rolling launch strategy, incidentally -- gives him time to collect data for the target sites and resolve the inevitable Analytics Is A Hard Problem (TM) issues, then the black magic algorithm will have some seed data for when it gets started.)
Genetify (<a href="http://news.ycombinator.com/item?id=376331" rel="nofollow">http://news.ycombinator.com/item?id=376331</a>) is one system that was recently open-sourced. A good discussion about it is on John Resig's site:<p><a href="http://ejohn.org/blog/genetic-ab-testing-with-javascript/" rel="nofollow">http://ejohn.org/blog/genetic-ab-testing-with-javascript/</a>
Would like to think that my comment here inspired this post: <a href="http://news.ycombinator.com/item?id=433084" rel="nofollow">http://news.ycombinator.com/item?id=433084</a> :)<p>As mentioned there, there are a couple of commercial products besides Google Website Optimizer, one being Optimost (now part of Interwoven). It does a lot more than just simple AB testing but is quite expensive (we were quoted around $10k/month when I played with it at my last company).<p>Genetify looks pretty cool and I'll spend some time playing with it. One thing I wish all these frameworks had is hooks for testing things server-side rather than with client Javascript. This would make it a lot easier to test process flows (i.e. "one-page form versus wizard") in addition to individual page elements.<p>If there are other open-source options out there, I'm looking forward to hearing about them.
Here's an ab testing plugin I wrote for rails: <a href="http://github.com/flyingma/rails_ab_split_tester/tree/master" rel="nofollow">http://github.com/flyingma/rails_ab_split_tester/tree/master</a><p>I'm hoping going to get something similar working for merb soon.
<a href="http://exp-platform.com/" rel="nofollow">http://exp-platform.com/</a> is an A/B testing platform used internally at Microsoft that will be made available externally at some point.
The author mentions Google Analytics, but it's not entirely clear whether he means that to include Google Website Optimizer (www.google.com/websiteoptimizer). I just completed a project with it and it is quite well implemented. The biggest downside is that you have to rely on JavaScript and requests to third-party servers.
I built one years ago (2001ish) but never even tried to commercialise it as there are tons of patents on it, with relatively big players. Perhaps they all folded, I haven't kept up with it.