75

A/B tests

A/B testing lets you test different versions of email, push notifications, and SMS messages in your campaign, so you can measure which version performs better. For email, this means you can test subject lines against each other, different versions of your call to action, or try out new designs. For SMS and push notifications, you can test for content, imagery, or even deep links.

You can test multiple messages in every triggered campaign, though we do recommend sticking to making just one change in your variation. This allows you to get an accurate picture of how that change measures up against the original without being confused by other data points.

How do I set up an A/B test?

For either email, Push, or SMS messages, this process is the same. First things first, you’ll want to head to the campaign where you want to run your test, and create the type of message you want to test—email or push notification.

Then, on the message you’d like to add a test to, you’ll see a Turn into A/B Test button.

By default, 100% of the traffic goes to the original and 0% to the variation.

Make your changes to the variation. You can change most anything about the message. In the case of email, this could be your from address, subject, body, even the sending mode (“Queue Draft” vs. “Send Automatically”). For push, feel free to test your push title, body content, or get creative with custom payloads. Any delays and time windows are shared across both versions, though.

We start routing traffic as soon as your campaign is live.

If you have a low volume of messages being sent or are just starting with A/B testing, try testing a small subject line or content change, and measuring opens.

Check in on your changes

Once you have an A/B test running in an active campaign, you should see an A/B Test tab appearing in your campaign overview. That’s where you can go to see the results of the campaign and pick a winner.

When you have statistically significant results, or if you want to end the test before that, you can pick one of the options as the winner by clicking the “Select winner” button. We’ll remove the other option and remove that particular A/B test from the screen.

Understanding your test results

Want more information about what statistical significance means or how we’re calculating different things? Read about it in more detail here.