<p><pre><code> During this time there were a total of 67 new
subscriptions. Of these 58% (39) came from the
new design and 42% (28) came from the old design.
Looks like the new one is a clear winner.
</code></pre>
Is it? This seems a small population to settle on a clear winner.<p>Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it).<p>I think this means that in a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time.<p>An alternative is the Adjusted Wald method. You can try it online here:<p><a href="https://measuringu.com/wald/" rel="nofollow">https://measuringu.com/wald/</a><p>Which gives some confidence intervals which also range from "could be better" to "could be worse". Even when you reduce the confidence level from the typical 95% to 90%.<p><pre><code> a quick check with an A/B testing calculator
even says that this result has significance
(~90% likely)
</code></pre>
Which calculator was that?
It's definitely not just the design itself. The CTA (your email submit) of the new landing page is better positioned throughout. You have also added urgency ("time until next shipment", "subscribe now before time runs out"). You can also see up to 3 previous candy boxes now, whereas before, you'd have to click through. All of those contribute to the conversion rate of your new landing page.<p>Your old landing page actually contains some great elements that are missing from the new one, like the reviews, the explanation anime video and the "new tastes only available in Japan", which seems a great feature to me. I'd definitely try and add those back in, and split test that new version against the current winner.
Both pages would probably convert better if there was an explanation as to what happens when I put in my email address. There's also a confusing use over the word 'mailbox' - which I'd use for email, but also physical mail. The page asks for an email, immediately after saying "candy surprises in your mailbox" - if you assume no previous knowledge of what this site is going to offer, it's pretty confusing. And if it's confusing, you're less likely to get people to sign up.<p>Without having a proper look at and understanding the site, I'm not sure whether that email is a leadgen form, or start of the sign up process, but either way you could probably increase conversion with some better explanatory copy.<p>Interesting read though! Thanks for sharing!
I think the actual product images are more tangible to the customer compared to the illustrations. My gut feeling is that the new design and repositioning of the elements on the page don't make a huge difference aside from drawing attention to the product images.
I don't think this is necessarily the right conclusion.<p>Imagine a friend with a cafe, they get the opportunity to get another cafe, same area, same clientele. With the new cafe they decide to just get some franchise, a Subway type of thing. People are familiar with it and flock there in relative droves. The original 'ma and pa' cafe is therefore deemed to be not as good as a 'template'.<p>The other option to the franchise could have been to have gone with the 'ma and pa' offering, same deal.<p>But that is not all of the choices. You could actually create something new, not hold with what you have or chuck it in and go with the 'franchise'.<p>With website design it is very easy for people to chuck it in and go with a Themeforest effort, Shopify, Squarespace and the like.<p>But a lot of 'new ingredients' have came along with CSS Grid, semantic markup and much else that the themes, 'serviced websites' and the like are not up to speed with yet and show no signs of wanting to implement.<p>It is also possible to build out more than just a landing page from scratch in two weeks with the new tools. This involves learning instead of botching someone else's floats, divs and margin hacks. We have Cargo Cult programming and in my opinion this story is just another example of this.<p>I would recommend anyone else in this position to start content first, not buying a theme and then taking the photos needed to push in the theme because the boxes are there already. This is a backwards, 'design led' process and there are plenty of good reasons to go content first, then structure it properly, spend a day or two with CSS grid, then put the existing branding on there. A static HTML page is a good start, a working prototype instead of a PDF mockup.<p>Then, instead of the A/B testing, ask an honest person who knows a thing or two and won't bullshit you. They will ask 'why have you done that' questions which will get the content in shape. Thereafter, once you have a go at doing it you can spot what you like elsewhere and learn from it rather than cargo-cult-copy-paste it to never be confident of anything.
Willing to bet that with some improvements to the original design the results would've been even better.<p>For example:<p>– Your navigation at the top is illegible. Theres very little contrast and it's hard to read. Make that more clear, higher contrast.<p>- Your main CTA ("Japanese candy surprises in your mailbox twice a month." [Email]) is also hard to read because of the orange and the custom font. Swap out the orange for a more vibrant color (maybe the orange color of the button) and use a nicer font that's easier to read.<p>- MAKE THE FONT SIZE BIGGER<p>- Add in some of the elements you added in the new design: When the next box is going out, etc.)<p>Your original design had an air of authenticity and honesty to it. The bought template seems like a copycat.<p>Keep (and improve!) the original!
Small sample size aside, they used the wrong KPI (imho). This needs to be tested over time. That is, which one retains better? Focusing on conversion's is a fool's errand. That is, you convert better but then if churn is higher you could end up at net loss. That's not ideal, obviously.<p>I think also, you'd have to look at referrals (if you have them). Perhaps a lower subscription rate actually led to more sales because (for some reason) those subscribers like to tell their friends.<p>The analysis here is too shallow.
The author lost credibility by describing statistical significance as unecessary for making informed decisions.<p>In this context, that’s just not correct.