I love the passion here, and I love Optimizely (we just recently signed up with them). Keep running you're bound to end up somewhere awesome. There are a few key points here that make this latest test you're running flawed though (and I know from running my own 120 multi-variate test).<p>Running all of those tests in a single run will help you find a test that improves over the original almost definitely. What that large test doesn't tell you definitively is how those variations perform against one another nor does it tell you the likely increase of your measured metric.<p>Furthermore, running tests without adherence to a rigorous thought process (where each variation essentially is a separate hypothesis, with possibly unique covariates, and different reasons why your test might be flawed.<p>Your own intuition and gut need to play a part in the tests so that you can understand the results and so that they don't cause more confusion.<p>In the end, you will end up with results that you could've gotten to 50x faster by just testing the variations you believed had a reasonable cause to do better (just a couple rather than 1000). This is again due to the fact that Optimizely is only telling you that it's likely the winning variation would do better and the fact that the individual results don't compare to anything aside from the original page.<p>I'd love to chat more and share my love for data and the knowledge I've gained as I've traveled this exact path with you. Reply if you wanna email/skype about this a bit.