50

Understanding your A/B test results

Note: the information in this document only applies to testing campaign messages. See A/B Testing Newsletters for more on testing newsletters.

What does ‘Chance to Beat Original’ mean?

Chance to beat original (CTBO) is the likelihood that the variation will outperform the original.

It determines statistical significance by answering the question:

Is the difference observed between the original and variation greater than a difference due to random chance?

What does a CTBO of 50% mean?

A CTBO of 50% means that the variation will outperform the original 50% of the time, which is the same as random chance. Therefore, there is no difference between your original and variation.

The closer your CTBO is to 50% (e.g., 40% or 60%), the less significant the difference between your original and variation.

The further your CTBO is from 50% (e.g., 5% and 90%), the more likely it is that there is a true difference between your original and variation.

What does “Original is beating variation” or vice versa mean?

Customer.io uses a significance level of 95%, meaning that CTBO has to be less than 5% or greater than 95% in order for us to report that one version is outperforming another.

If CTBO is > 95%, you can infer that your variation is outperforming your original.

If CTBO is < 5%, you can infer that your original is outperforming your variation.

What does “Not significant, need more data” mean?

If your CTBO is between 5% and 95%, it does not meet our threshold for statistical significance. If your sent volume is low, gathering more data may prove that a significant difference exists.

How is ‘Chance to Beat Original’ CTBO calculated?

There are 3 steps to calculating CTBO:

1. Calculate the standard error for the original and variation.

x = number of messages with the desired behaviour (e.g. clicks)
n = proportion of all messages with the desired behaviour (x / total, between 0 and 1)
standardError = SQRT(( n * (1 - n) / x ))

2. Calculate the Z-score for the original and variation.

p_o = mean success rate for the original
p_v = mean success rate for the variation
se_o = standard error for the original
se_v = standard error for the variation
Z-score = (p_o-p_v)/SQRT(POWER(se_o,2)+POWER(se_v,2))

3. Based on the Z-score use a statistics table to determine the p-value and CTBO.

CTBO = 1 - p-value