I am trying to optimize my adwords campaign, and running into the 'statistical significance' question.<p>I read this great post from Paras Chopra here:
http://visualwebsiteoptimizer.com/split-testing-blog/ab-testing-significance-calculator-spreadsheet-in-excel/<p>Though in my case I started my campaign with many keywords (about 400 keywords), and I now have the following:<p>Keyword group A (10 keywords - the "best"): 47 clicks, 16 conversions<p>Keyword group B (390 keywords - the rest): 244 clicks, 18 conversions<p>If I apply these data into the A/B Testing Significance calculator, I obtain a 99% confidence that A is better than B.
But does it really work with a group of keywords, or is this calculation meant to be used only for strict "singular" test cases?
You can use it this way. But what does the data tell? Group A is definitely performing better than group B or better it is not a random coincidence that group A is better performing.