ExactTarget A/B testing: are email domains split evenly?

While setting up an subject line A/B test in ExactTarget, a potential issue dawned on me: What happens if one of the test groups has a much larger share of Gmail addresses? As we know, Gmail downloads images on it’s servers automatically and therefore those email addresses will count as opens. If one test group had a disproportionally larger share of Gmail addresses, then the results would be skewed.

However, I’m happy to report that this wasn’t the case.

After the A/B test went out, I exported the two separate lists. For this test, there were a total of 94,0002 email addresses.

Here’s the breakdown of the more popular email domains:

Gmail: 12,885 vs. 13,026

Yahoo: 8,818 vs. 8,847

Hotmail: 4,377 vs. 4,321

Gmail still had a 141 difference, which could cause a potential swing of 0.15%, but this is not significant.

Leave a Reply

Your email address will not be published. Required fields are marked *