Every time they say "classic statistics" just insert "what we did before now" and see how frustrated you get with this announcement. The whole point of using them is that people don't need a statistician because the tool should make it easy to run solid tests. That of course hasn't been the case and they're finally admitting it.
I wrote about the problem with sequential testing in online experiment three years ago on the Custora blog [1]. And Evan Miller wrote about it two years before me on his blog [2]. I'm glad to see Optimizely finally getting on board. Communicating statistical significance to marketers is always challenging, and I'm sure this will lead to better decisions being made.<p>[1] <a href="http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-testing/" rel="nofollow">http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-te...</a><p>[2] <a href="http://www.evanmiller.org/how-not-to-run-an-ab-test.html" rel="nofollow">http://www.evanmiller.org/how-not-to-run-an-ab-test.html</a>
What a misleading comparison: <a href="http://i.imgur.com/aWqxk2U.png" rel="nofollow">http://i.imgur.com/aWqxk2U.png</a><p>"95% chance you'll make the <i>right</i> decision" vs "30% chance you'll make the <i>wrong</i> decision", emphasis mine.
This graph is a joke: <a href="https://d1qmdf3vop2l07.cloudfront.net/optimizely-marketer-assets.cloudvent.net/raw/statistics/sequential-testing-graphic-us.png" rel="nofollow">https://d1qmdf3vop2l07.cloudfront.net/optimizely-marketer-as...</a><p>And any company trying to sell a statistical tool/package that would actually create a graph like that is selling snake oil. Your model only gets better, digitally, and never sees a regression? And you're using this for <i>web analytics?</i>
The claimed math behind the 'new engine':
<a href="http://pages.optimizely.com/rs/optimizely/images/stats_engine_technical_paper.pdf" rel="nofollow">http://pages.optimizely.com/rs/optimizely/images/stats_engin...</a>
I'm someone that would consider using Optimizely: no formal stats background but understand high school stats, work on web apps professionally and interested in analytics and testing. I've watched the video, read everything on the page and I still don't understand what they're trying to tell me here.<p>Based on my admittedly limited understanding of stats, unless you set the sample size and decide what significance is in advance your test will probably misinform you. Nothing on this landing page explains to me how this new thing might mean otherwise and it really doesn't help that the page is otherwise full of hubris, eg:"goodbye traditional statistics". Somehow it seems unlikely that a web startup just invalidated all of statistics
For those who don't want to enter personal details, PDF is on this URL: <a href="http://pages.optimizely.com/rs/optimizely/images/practical_guide_to_stats.pdf" rel="nofollow">http://pages.optimizely.com/rs/optimizely/images/practical_g...</a>
I am surprised by all the negative commentary here. On the whole, companies like Optimizely, RJMetrics, Custora, and others are doing more to push statistical analysis to the mass market than anyone else. These tools are not designed for statisticians or ML practitioners so it makes sense they do not put language like Bayesian, etc. front and center. IMO, the more people using data to make decisions, the better.
Interesting that Optimizely is positioning themselves as the over arching discipline as "Statistics reinvented for the internet age". My guess is to parry against the onslaught of A/B testing and optimization platforms for web and mobile from all directions. Of course with their stable of PHD statisticians and data scientists, Optimizely is the answer.
Fantastic to see Optimizely changing their stats model. The more education that is done on web experimentation the better as there certainly still is a lot of snake oil being sold out there!<p>Their chosen technique is one way of solving the problem of communicating statistics to non-technical audiences however the interpretation of the results may suffer here. I can imagine that this technique will lead to overestimations of the effect size in situations where the threshold is reached early in an experiment as it will reward extreme values observed when the experiment is under powered.
a lot of people are denigrating the site based on its content, but i'll, fully expecting to be down-voted, go out on a limb and say what I think we are all thinking: "i just don't like that guy's sweater".