A/B Testing the Effect of Sender Race on Email Response Rates

You spammed a million people for this? Enough that over a hundred thousand people spent the time and energy to write an email reply? Please do justice to the collective time you took up with the survey.<p>Could you break down the data by name and/or gender of both the sender and the respondent? Test multiple hypotheses simultaneously? For example, in page 1002 of the referenced research article (<a href="http://faculty.chicagobooth.edu/marianne.bertrand/research/papers/emily_lakisha_aer.pdf" rel="nofollow">http://faculty.chicagobooth.edu/marianne.bertrand/research/p...</a>), they break down a whole slew of other qualities that may have affected response rate.<p>It's entirely possible that the name choice for your study has more of an effect than the perceived race association of those names. The "99.9% confidence" cited is really not that, and is subject to various biases which are hard to discern.

I get annoyed when I read things like: "A performed 4.9% better than B", when the comparison is between two proportions.<p>Just say it like it is, without putting a slant on it - A's response rate was 0.6% higher than B's.<p>I don't care if the results are significantly different, if the difference between the two samples is so small.

Eric, the write-up is great, but it would be better if you provided the supporting data. The research is interesting, but sending out a million emails with tracking bugs is a bit, umm, questionable when one considers the time/effort wasted by the recipients. If everyone ran similar experiments, it would make a real mess, so providing the data you collected could also be beneficial in reducing the load.

Applaud you for doing the research, and I'll probably end up referencing it in my work at some point. Surely there are a number of holes to poke (as with anything) the one that stands out to me at the moment is you didn't control for the race of the recipient. I.e. if your overall recipient list was 50% Hispanic, even though you randomized who got what, would still expect Hispanic-sent open rate to be higher.

op here. i should own up. this is an apology.<p>@dantillberg, @jcr, et al: you're right.<p>i heard about the original study, was curious, but not enough to think much of it. in a previous startup we had a female intern who was getting substantially better response rates than the male founders.<p>after the recent press about how women in senior roles correlates with startup success i became a bit more curious and wondered if i could craft the perfect "from" field for outgoing emails.<p>i admit this was aggressive and that I got carried away. that's no excuse.

I assume the author was just trying to set up the premise before getting to the meat of the article but the opening paragraph claim that Americans are more sensitive about discrimination than anybody else is probably false and more than a little ironic.

File this one under "tests that shouldn't have been conducted in the first place"

You should have also included unconventional names not usually associated with black people, such as Moon Unit, Starshine, Whalesong, and other such "hippy" names, or names associated with poor rural white people.

Offtopic but I did a double take on this: "Former options trader. <i>Passionate QBASIC developer</i>." Subtle irony or what?

Frankly, I'm not really surprised that Antonio Banderas commanded a higher response rate.