Which is why scientific tests have a placebo option.<p>Very loosely, what hypothesis testing is telling you is whether your results are so aberrant that they can't just be explained by model variance (coins flipped at random). The hypothesis that can't really be tested is whether your model explain most of the variance in observed data.<p>If I take a random sample of people, ask them whether they're male or female and what's their yearly earnings, I'll find statistically significant differences in averages; however, that will explain only a very small portion of the total variance in wages, given that these also depend on education, skills, experience...<p>It could be that the main source of variance in the poster's experiment is the time of the day emails are sent. How would he know? When doing crude A/B testing one hopes that all the things in the world one _isn't_ controlling (is it raining? have local sports team won trophies?) are uncorrelated with the control variable. But here it might not, there might be something very different between the batches he doesn't know about.<p>And he's proud of that!