Optimizely’s decision to ditch its free plan

A&#x2F;B&#x2F;n testing is certainly not dead. It&#x27;s more likely Optimizely failed to see enough movement from their free plan into their paid tiers and in an effort to streamline&#x2F;focus&#x2F;cost-cut they&#x27;re dropping the free plan.<p>This is interesting because Optimizely was in-part responsible for commoditizing A&#x2F;B testing in 2010&#x2F;11. Previously, it was a thousands-of-dollars per month product.

When I started Optimizely with my co-founder, Pete Koomen, our goal was to enable anyone in the world to do website A&#x2F;B testing. I believed then, as I do now, that we are democratizing the scientific method. That said, when a company doesn’t consider experimentation a business critical process, they are unlikely to dedicate the internal resources necessary to make it successful. We experimented (ha!) with a free plan to see if it would be a good model for getting customers to dip their toe into A&#x2F;B testing. Turns out, the data showed it wasn’t as effective as a robust free trial, which we continue to offer for all products.<p>Also, A&#x2F;B testing is far from dead. Companies that have a culture of experimentation have grown their market cap by 756% in the last five years. See our Opticon keynote for more context: <a href="https:&#x2F;&#x2F;www.optimizely.com&#x2F;opticon&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.optimizely.com&#x2F;opticon&#x2F;</a>

We were using Optimizely until we didn&#x27;t qualify for the lowest tier paid plan. Not enough impressions?<p>We knew at our traffic tier, Optimizely experiment analysis was not going to be statically robust. We were going to have our data person do the analysis with different techniques to deal with the smaller sample size. What we really wanted was the interface for a non-technical person to create the variation and Optimizely to manage which variation people saw.<p>Instead of letting us continue, our account was canceled. In the next months, when we do have the traffic, we are not going to consider them. Not impressed.

It seems far from dead, there are thousands of sites (many enterprise) that are currently using Optimizely&#x27;s js library<p><a href="https:&#x2F;&#x2F;nerdydata.com&#x2F;technology-reports&#x2F;optimizely&#x2F;78649239-9bbe-4383-a09c-f65e072e54d8" rel="nofollow">https:&#x2F;&#x2F;nerdydata.com&#x2F;technology-reports&#x2F;optimizely&#x2F;78649239...</a>

I get it, the economics of the web force articles to have silly clickbait titles like this one, but it&#x27;s still annoying.<p>I thought the article actually did a good job of exposing the problems of &quot;naive&quot; A&#x2F;B testing, but the conclusions of the article do not say, at all, that &quot;A&#x2F;B web testing is dead&quot;.

Optimizely presumably has no incentive to compete with Google Optimize for free users: <a href="https:&#x2F;&#x2F;www.google.com&#x2F;analytics&#x2F;optimize&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.google.com&#x2F;analytics&#x2F;optimize&#x2F;</a> .

As I&#x27;ve been saying for several years, most website A&#x2F;B testing in the wild is statistically illiterate, expensive nonsense. Optimizely is right to cull their MVT product.

I&#x27;m curious if there will be any negative upstream effects in this decision down the road. Optimizely doesn&#x27;t need to be free, but an affordable and mainstream option seems smart t get early-stage companies in the door. A wise competitor would take advantage of this new gap in the market.

Author of the article here :)<p>I&#x27;m glad to hear it sparked your interest (btw, I didn&#x27;t pick the headline).<p>I agree with what has been said about A&#x2F;B testing so far - it is a great tool for more mature companies, but I hope I could also make my case for personalization. After all, the &quot;let&#x27;s test that&quot; mentality is not always applicable. Startups and SMEs should not underestimate how good their gut feeling is when it comes to optimizing their website for different audiences. And yes, splitting your visitors into different segments might seem counterintuitive when talking about testing but I strongly believe it&#x27;s the way to go for companies who haven&#x27;t achieved their desired results with A&#x2F;B testing.<p>Disclaimer: I work for Unless, a personalization service.

SplitOptimizer.com is a very cheap a&#x2F;b testing tool. It’s not as full featured as optimized but it works perfectly.

In addition to scrapping a free plan, they also have moved away from displaying pricing ALL TOGETHER.

So A&#x2F;B testing isn’t dead...just doing it the wrong way is slowly dying, maybe.

Are there any open source alternatives to Optimizely? If not, would an experienced SWE comment on an estimated difficulty level to get an MVP together?

CEO of VWO here - a competitor to Optimizely.<p>The article&#x27;s title &quot;A&#x2F;B testing is dead&quot; is very misleading. A&#x2F;B testing is not a tool, it&#x27;s mental model seeking learnings and improvements. Anytime you introduce a new change to your business, you&#x27;re implicitly or explicitly trying to understand its impact. Most of the times, setting up an environment to tease out real effects of change is hard (experiments in strategy are usually very difficult) so you&#x27;re left to guessing.<p>However, for mediums that allow easy tests (like website, landing pages where you can set up scientific experiments with ease and have enough traffic to get results in a short time period), why wouldn&#x27;t you? The way I see it, rejection of A&#x2F;B testing is the rejection of the scientific method of conjecture and improvement. In the highly contextual business environment with tens and hundreds of conditional variables, best practices and &quot;gut feeling&quot; can only go so far.<p>A&#x2F;B testing for very small businesses (those who get attracted to free plans) doesn&#x27;t make a lot of sense because they usually don&#x27;t have enough traffic to get statistically significant results. Plus, optimization via A&#x2F;B testing comes only after you&#x27;ve gotten your basics right and it&#x27;s now time to scale. This is why at VWO we never had a free plan.<p>So Optimizely&#x27;s decision to ditch free plan makes sense.