It was three months ago to the day that we announced our private beta. We were thrilled to see thousands of people sign up for the beta. We've worked hard to improve the product over the last three months based on their feedback and we're now ready to come out of beta and launch to the public!<p>Our goal is to make it as easy as humanly possible for you to create and run A/B tests on your site. All you need to do is enter your website URL and point and click on what you want to change. Absolutely no coding or engineering required. You don't even need to create an account to get started.<p>We're working hard to improve the product every day based on your feedback-- just reply to this comment or shoot us an email at feedback@optimizely.com and let us know what you think!
You might want to adjust the pricing tiers. With $19/$79/$399 per month tiers, I have to guess that most people will start by trying the lowest tier; but that's limited to 2k visitors.<p>If you're doing an A/B test with 2k visitors and you've got an "A" conversion rate of 1%, you need to see a 60% improvement (16 conversions vs. 10) in order to have statistical significance. That's a huge improvement, and I doubt many people will be that lucky.<p>You'll probably be better off setting the limit for the lowest tier at 10k (and adjusting the others upwards too); that should dramatically increase the number of new users who find your service useful and stick around.<p>EDIT: Or another option would be to have a "free 10,000 visitors" trial rather than a "free 30 days" trial.
Disclaimer: I have watched the video but not tried the product, so some of these observations may not apply to the the product itself.<p>Overall, this looks like a great product. I know a number of people who find Google Website Optimizer complicated to use, and I would definitely recommend this to them as a simpler option. I love how slick the browser interface to edit pages is, and I think having the default 'engagement' metric so people can see results without having to set up a goal page is a brilliant idea.<p>There are a few things you have done which I would consider doing differently.<p>1. It looks like your mission is to make A/B testing really easy, but your pricing page at the moment doesn't really reflect that. Number of visitors tested is an easy metric, but one that it is hard for me to interpret without lots of knowledge of A/B testing. How many tests does this mean I can run how quickly?<p>I would also reconsider the additional features you offer in premium packages. Cross-browser testing sounds complex and makes me worry that your site edits will fail in IE6. I don't want to have to test it, I just want it to work. With uptime monitoring, what does this have to do with A/B testing? Bigger sites probably already have some form of monitoring already anyway, so it looks like they are going to pay for something they don't need. I think your core product is strong enough that you don't need to offer these.<p>2. Showing me the percentage significance level appeals to my inner stats nerd, but I suspect the sort of people I think will benefit most from Optimizely will have difficulty interpreting this number. What level is 'ok'? Having a rank out of 5 below doesn't really address this, is 4/5 ok or do I need 5/5? Google deal with this very well with their bars which turn red or green when they reach significance.<p>3. The 'select container' option to expand the selection seems non-obvious, and isn't how multi-select works in any other interface I've seen. Maybe allow people to select multiple components and then take their deepest common parent?<p>There are also some additional features I personally would like to see<p>1. It would be great if you gave an estimate for how long until my experiment will reach an appropriate significance level (obviously based on % change and traffic seen so far).<p>2. I would like to be able to choose my conversion action by clicking on a form button or link in your page editor.<p>3. It would be amazingly useful to have some automatic suggestions for how a page could be changed - on many occasions I've seen people resist A/B testing because the options are so wide and they don't know what to do. Doing this for some simple suggestions sounds possible - e.g. making key links bigger and moving them up the page. Doing anything more sophisticated could be a good challenge though :-)
It was three months ago to the day that we announced our private beta. We were thrilled to see thousands of people sign up for the beta. We've worked hard to improve the product over the last three months based on their feedback and we're now ready to come out of beta and launch to the public!<p>Our goal is to make it as easy as humanly possible for you to create and run A/B tests on your site. All you need to do is enter your website URL and point and click on what you want to change. Absolutely no coding or engineering required. You don't even need to create an account to get started.<p>We're working hard to improve the product every day based on your feedback-- just reply to this comment or shoot us an email at feedback at optimizely dot com and let us know what you think!
What's the USP on Optimzely when compared to the already quite formidably awesome Visual Website Optimizer?<p>Also, as someone that really wants to incorporate this tech into my larger scale marketing efforts I am put off by the pricing models to both of the above mentioned sources.<p>When only one of several ad campaigns are doing 100k visitors a day you can see how that pricing model does me no good. Particularly when you take into consideration that I never stop testing -something- on a campaigns landing page.<p>Also would love an API so that something like this could be tightly integrated into my custom conversion analytics solution.<p>Congratulations on your launch nonetheless.
Can someone A/B test a web startup's name with words ending phonetically as "ly" or "er" versus a more "normal" name?<p>Kidding aside, the product looks really great - congratulations on launching!<p>For a critique, I'll say that the intro video is very dry. A/B it with something less Ben Stein?<p>One thing I'm always interested in is pricing. How did you go about determining your various plans and rates?
I don't understand people <i>using a CMS</i> thinking that Google Website Optimizer (GWO) for A/B tests is hard<p>I got myself a plugin for WordPress that lets you insert in any page the GWO javascripts
I created a variation page on WordPress, identified the original one and the conversion page.
On the site i created the test, started it and it's just working.<p>But even if you're not using a CMS, copy/paste some code at the beginning and end of a HTML page? how is that so hard?<p>Edit: clarified
Really cool A/B Testing tool :)<p>Your variation editor looks really awesome too. How much effort does it take create something like that? I mean just the HTML hover/manipulation etc. I am planning to write a scraper related tool similar to <a href="http://open.dapper.net/" rel="nofollow">http://open.dapper.net/</a><p>Did you encounter any serious cross browser issues? Or any other notable info you would like to share with the HN community. Is there any open source basic implementation that can be used as a starting point? Do you preprocess the page on the server or is it all Javascript magic on the client side?
Suggestions:<p>In a couple of spots, the edit HTML dialog overlaid the items I was trying to edit. I attempted and expected to be able to move the edit dialog around the page and out of the way of what I was editing.<p>On my own site, I was particularly wanting to play with colors/themes. Not sure what the best way to implement this is, but the only way I could do that through this interface was with style tags.<p>Awesome product! I will likely use it in the near future.
This looks brilliant, you just need to nail your pricing structure.. perhaps consider per test or limit the number of tests and variations? Either way, congratulations, some serious skills on show form your engineer(s)!
Great stuff, the video made me want to jump right in and test stuff; too bad I don't have any medium/big sites to test :)<p>For us folks with small-traffic websites (ie. your Bronze plan), an useful addition would be a suggestion to people to not try too many variations at the same time. (I don't have the math, but I assume less variations means less datapoints until you get significant results)<p>Perhaps, as another commenter suggested, by estimating how long (in time or visitors) it will take - and by showing this estimate <i>while</i> adding variations.
nice idea. I tried to setup a new a/b test for a site and it became 'not easy' when I wanted to track an adsense click as a goal.<p>I wrote a manual a/b test for the same site about 7 years ago, and got adsense click goals working with 40 variations of the page, after about a week of working on it after work at night after my regular job.<p>There is some jquery code you can add to track javascript events which would probably track adsense clicks, but it's 'not easy', ie: I might as well not pay for it.<p>Cool idea though! I would pay for it if I could track adsense clicks without having to write code. Also, having an option to select multiple dom elements at once would be nice. In addition, logging into the web site and editing the html to add the code in the <head> tag would be nice too, especially since it's kind of a hassle for me to open eclipse and re-deploy to app engine at 1am when I am tired.<p>haha, hope this helps.
Do any of these A/B testing tools support offline conversions in any meaningful way? For us a phone call is a conversion. I'd love to be able to build the logic to support that once (give me an API call along with GET parameters to the page that I can post back to you), and then be able to use tools like this to actually try different tests.
Great job on the product and video - really easy and slick.<p>What's the use of real-time results for A/B testing? Most likely, it's only going to lead to erroneous conclusions, as discussed in this thread - <a href="http://news.ycombinator.com/item?id=1778104" rel="nofollow">http://news.ycombinator.com/item?id=1778104</a>.
This website is great. I was able to jump in and mess around with my website's home page within seconds after watching the video. I'm not a developer and being able to help out with A/B testing will be a huge help and one less thing our developers will have to deal with. Looking forward to using it!
This is great, just in time for my testing :), I'll definitely give it a try! I would love for one of your pricing packages to include speaking directly with an optimization consultant, or perhaps receiving a detailed report that includes optimization recommendations.
Why do all of these companies create cookie cutter names like 'Optimizely' is '<i>ly' the next .com?<p>To me </i>ly says 'We have no track record, use with caution' I would feel more comfortable using something with a name that isn't following some online fashion.
Optimizely sounds like a great service but after entering my URL it took a long time to load. It would be great if you could speed that up... If I hadn't heard of the link from Hacker News, I would have closed the tab after a few seconds.
I get the impression that home page has been optimised to within an inch of it's life and converts like crazy. A very impressive landing page, I had the overwhelming urge to put in my Url and try it out. Well done.
Will it be possible to define URLs to different, already existing versions (as google does)?<p>Your online editing is awesome esp. for small things but sometimes one might be able to render already existing pages…
This is a company of really smart and talented people. I have had limited experience with the beta, but have been following the project with much interest. I expect great things from Optimizely.
Just gave it a spin and it's a really cool product - feels a bit buggy at the moment but for the right application this would be a really valuable tool to have. Great job, guys.