Results from Candy Japan box design A/B test

In a past life I founded&#x2F;operated a conversion rate optimization company.<p>One of our all time most successful tests was related to this story and happened almost by accident.<p>The client had just rolled out new packaging and wanted help validating the change statistically, very similar to this situation.<p>Their metric was measuring upsells to a premium product after a purchase, and like Candy Japan, found statistically insignificant results.<p>But the new packaging was beautiful and we wanted to test it on their landing page.<p>We tried just replacing the old product image on landing page. Meh results, no real bump.<p>We tried increasing the size of the image + circling with &quot;Improved&quot; (no joke). 10% bump. Interesting.<p>We hired a professional model to hold the product and smile. 15% bump. !?<p>We hired a group of professional models to hold the product and smile. 40% bump in sales. !@%K$!?<p>We re-hired the same group of people to hold product, look SUPER excited (jumping up and down like their team just won the superbowl). 50%+ bump in sales.<p>No joke.<p>If Candy Japan is reading and you&#x27;d like some help recreating I would not be shocked to see something similarly group oriented, visceral, and beautiful bump yeh.

How about a plain box in some relevant color (sakura blossom pink? :) ), hand-stamped with the Candy Japan logo?<p>In my mind&#x27;s eye, that&#x27;d remain classy enough not to be creepy&#x2F;cringy&#x2F;embarrassing (as <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11641715" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11641715</a> complains), but would also have sort of a personal touch to it.<p>Having a couple rubber stamps made and stamping each box during packaging wouldn&#x27;t probably cost an awful lot...

Have you considered that the new box is too &quot;generic&quot; or &quot;corporate?&quot; (For lack of a better word)<p>In other words, the old box, while plain, adds a certain intimacy. It doesn&#x27;t look like a huge multinational candy company sent it. The new box, though, looks basically like any other piece of Japanese media.

I much prefer the plain packaging because:<p>- The anime picture is embarrassing to receive in the mail &#x2F; at work.<p>- The picture itself is kind of creepy, someone walking past my desk once even described it as &#x27;rapey&#x27;.<p>- I can reuse the box.<p>- I am not 6.

I run a subscription business through mail myself, and I love your posts.<p>Have you tried to run introduction offers? 50% off for first shipment for example? If I were you I would try to put the countdown for the next shipment on top of your site. Thats a really powerful psychological factor.<p>Your CTA is also a little unclear, why do I need to first get an email, then a link? The buy process should be straight forward, not through my email. And if you have a really good reason for doing it this way, you should atleast tell me why I need to leave my email.<p>I really like the cartoon that explains the process. Very good.<p>Regarding packaging design, we did a huge remake in 2013. We increased our prices with 30%, and redesigned packaging completely. Doubled our LTV, insane gains. Just keep testing, but try to change provider for carton. We pay $0,1 USD for bigger cartons then yours, in one of the most expensive countries in the world. (With plastic window for address field) We chose to put our three USP on the back of the shipment box, to remind your customers every time they receive a shipment why they subscribed in the first place.<p>Keep testing!

I don&#x27;t really think A&#x2F;B testing something like this, with this size of a sample, on a subscription based niche-product can give you any real insights.<p>Why not poll your subscribers or (assuming you can get the costs to work out) have an option for anime art. Personally I wouldn&#x27;t want the anime style box, something custom and simpler would be fine.. but thats my personal opinion.

&gt; In a world where it makes no difference which package is sent, you would get a result as significant as this 80% of the time.<p>I suddenly get p-values now. Why has no one ever explained it like this before?

Just a small update, but I just ran a survey for customers to see which package they would rather have.<p>Which package design would you rather receive?<p>The white one with illustration 79%<p>The plain dark one 21%<p>Total votes: 52

I like my subscription. My biggest issue has been shipments that end up in my mailbox melted. It would probably not be possible for new packaging to fix this, but this would be an area of improvement worth higher cost.

I think candyjapan should add a new form of subscription, one-time shipment, and I believe they would see it being ordered by about 40% of their customers. ;)

One consideration is the user experience is not connectable to a purchase.<p>If I went to a friends house and saw this, I would never remember the name. If you could combine &quot;making the purchaser look cool&quot; with timing-relevancy, I think your packaging improvement would pay dividends.<p>Put another way, I believe your A&#x2F;B ratio test is actually X*Y&#x2F;Z where X&#x2F;Z and Y&#x2F;Z are both &lt; 1. Synergy.<p>Terrible idea, (possibly) useful for analogy: Instead of optimizing the unboxing experience alone, if you also included cards with QR codes, where every QR code immediately triggers a free candy bar (with user data entry on a mobile app). It should also provide social value to the host in realtime; maybe they also receive an immediate notification that their guest scanned their card plus some other bonus I can&#x27;t think of.<p>The point is to get closer to the conversion&#x2F;churn decision, or provide more value to your consumer.

I signed up for CandyJapan several years ago when they were just getting things started. I canceled after probably 3 shipments. Each shipment I was only hoping that the next would contain better products and actually be worth the value in which I was paying. Unfortunately that never happened. While a cool concept and very eye catching, that&#x27;s all it is. Importing candy is expensive no matter how you go about it. Most people just want a poke to see how it is, and then move on to something else.

Both packages look pretty ugly. Hire a better designer...

Could the customs sticker have caused problems?<p>I used to receive Candy Japan in the &quot;plain&quot; box to Australia, and the customs sticker declaring the value &amp; contents took up most of the space on the box.

Just a FYI: The pictures stopped working here: <a href="https:&#x2F;&#x2F;www.candyjapan.com&#x2F;mini-assort" rel="nofollow">https:&#x2F;&#x2F;www.candyjapan.com&#x2F;mini-assort</a>

Why not allow the customer to choose the box type and default to the plain box. Add $1 for the designer box, see what people choose?

Now you have to try out different designs to see if it matters (it does and it can).

Just out of curiosity, which statistical test should be used in this case?

That 30 day shipping period is a killer. Candy is an impulsive treat. :)

Honestly, I think it would have been really bizarre if something as unrelated to the central product as the design on the box, changed something as fundamental as retention of customers.