Unless a startup is doing something absolutely terrible, a 45% increase is just not likely. Your advice could better be rephrased as "if you are seeing low enough numbers that it effects the viability of your company, redesign X and then A/B test the change."<p>Startups have a hundred better things they could be spending time on. A/B testing preys on a founders sense that they are leaving money on the table and not optimizing for maximum profit, sometimes to the detriment of focusing on things like good hires or critical product changes.<p><i>Edit: It isn't obvious until I clicked around, but the author is a little biased and should disclose he is the "Head of Optimization" at Optimizely</i>
These types of articles should quote error ranges.<p>"Optimizely's homepage redesign that yielded a 46% '+/- x%' increase in new accounts"<p>They never seem to do so.<p>If you have small traffic, forget about performing A/B tests. Instead learn from the A/B tests performed by companies with high traffic, and implement their findings.
Incremental performance jumps with huge % increase wins are extremely rare.<p>I can easily jack up click-throughs on a new banner or get users to fill out forms more by changing text copy and/or CTA size/positioning/color, but that doesn't mean those increases translate to an incremental improvement in overall prospect to lead conversion rates - you very well might just be shooting yourself in the foot by killing good conversion rates later on in the funnel with your upstream changes.<p>A good example was moving the lead gen form on a paid search destination landing page from below the fold to above the fold.. what did we see? More form conversions, a 30%+ increase actually. Yay! Dug into it further and what we actually found was that customers who were looking for customer service were clicking paid search ads for the company and filling out the form to try to reach customer service rather than reading the lead gen CTA's on the page that clearly spoke to new customers.<p>Unless you have a very controlled linear sales funnel, websites are spider-webs with numerous paths visitors could take to arrive at the content they're looking for. So you always have to take into account what downstream changes you could be introducing with a "big win increase test" on your site's homepage.
It's funny, but I've noticed that all of my clients in this space are just outright opposed to a/b testing. Most don't even bother to leverage analytics at all. I think more than anything they see the level of effort to ROI ratio too high.<p>For example, a startup not testing small changes right? Well unfortunately, an entire landing page makeover, while not overly difficult can windup taking a great deal of time. The redesign alone is just such an "opinionated" process (even by people who know nothing about design...) that throwing that much investment just to test is hard to justify. So even though most startups shouldn't waste time on stuff like button colors...that's often the only thing they'd have time for.<p>Of course I'm talking about the smaller scrappy startups. Not the ones backed by prolific amounts of capital.<p>Don't get me wrong. This is a GREAT article, I just would like to know the best level of effort for ab testing for the smaller guys.