While setting up an subject line A/B test in ExactTarget, a potential issue dawned on me: What happens if one of the test groups has a much larger share of Gmail addresses? As we know, Gmail downloads images on it’s servers automatically and therefore those email addresses will count as opens. If one test group had a disproportionally larger share of Gmail addresses, then the results would be skewed.
However, I’m happy to report that this wasn’t the case.
After the A/B test went out, I exported the two separate lists. For this test, there were a total of 94,0002 email addresses.
Here’s the breakdown of the more popular email domains:
Gmail: 12,885 vs. 13,026
Yahoo: 8,818 vs. 8,847
Hotmail: 4,377 vs. 4,321
Gmail still had a 141 difference, which could cause a potential swing of 0.15%, but this is not significant.