Understand results for classic A/B tests
Note: the information in this document only applies to testing automation messages. See A/B Testing One-time sends for more on testing one-time sends.
What is statistical significance?
Calculating statistical significance is a way of determining how confident you can be that the results are explained by the differences in the variations and are conclusive.
For example, if you want to test whether one email subject line causes more opens than another, and variation A has a 60% open rate and variation B has a 40% open rate, then variation A is the more successful subject line, right? Not necessarily. Understanding a result requires looking at how many people have received each variation and how many performed the desired action.
Imagine that you only sent each variation to 5 people and 3 of the 5 opened variation A, while 2 of the 5 opened variation B. Is a difference of only 1 out of 5 enough to indicate that the opens were actually because of the subject line and not just due to chance? Definitely not.
Now imagine that you sent each variation to 5,000 people. This means that 3,000 opened variation A and 2,000 opened variation B. You’re starting to build a bit more confidence in the fact that the subject line of variation A is the reason that 1,000 more people opened A than B. It’s intuitively clear that the larger numbers provide more confidence, but how confident?
Ideally, you aim for 95% confidence in your tests to declare it valid. The more people interact with your messages over time, the higher your confidence level can be.
In marketing, where your primary goal is to get a message out to your people in a timely fashion, you can also choose to accept a lower confidence level so you can make a decision faster. The question you should ask is, “If I aim for an 80% level of confidence, can I really tolerate a 1 in 5 chance that my test is misleading me?” If nothing else, statistical significance can help your team determine which tests are worthwhile, and which can be skipped because you can’t get a high enough confidence level in the result.
Understand ‘Chance to Beat Original’
Chance to beat original (CTBO) is the likelihood that the variation will outperform the original.
It determines statistical significance by answering the question: Is the difference observed between the original and variation greater than a difference due to random chance?
What does a CTBO of 50% mean?
A CTBO of 50% means that the variation will outperform the original 50% of the time, which is the same as random chance. Therefore, there is no difference between your original and variation.
The closer your CTBO is to 50% (e.g., 40% or 60%), the less significant the difference between your original and variation.
The further your CTBO is from 50% (e.g., 5% and 90%), the more likely it is that there is a true difference between your original and variation.
What does “Original is beating variation” or vice versa mean?
Customer.io uses a significance level of 95%, meaning that CTBO has to be less than 5% or greater than 95% in order for us to report that one version is outperforming another.
If CTBO is > 95%, you can infer that your variation is outperforming your original.
If CTBO is < 5%, you can infer that your original is outperforming your variation.
What does “Not significant, need more data” mean?
If your CTBO is between 5% and 95%, it does not meet our threshold for statistical significance. If your sent volume is low, gathering more data may prove that a significant difference exists.
How is ‘Chance to Beat Original’ CTBO calculated?
There are 3 steps to calculating CTBO:
1. Calculate the standard error for the original and variation.
x = number of messages with the desired behavior (e.g. clicks)
n = proportion of all messages with the desired behavior (x / total, between 0 and 1)
standardError = SQRT(( n * (1 - n) / x ))2. Calculate the Z-score for the original and variation.
p_o = mean success rate for the original
p_v = mean success rate for the variation
se_o = standard error for the original
se_v = standard error for the variation
Z-score = (p_o-p_v)/SQRT(POWER(se_o,2)+POWER(se_v,2))3. Based on the Z-score use a statistics table to determine the p-value and CTBO.
CTBO = 1 - p-value