# Holdout tests

## How it works[](#how-it-works)

A *holdout* is a type of A/B or random cohort test for email messages. But, rather than testing how your audience responds to variations of a message, some members of your audience will receive a message, and some won’t—they’re purposely “held out” of the test. This helps you determine whether a message is useful or not.

When you setup a holdout test in your campaign, you’ll determine the percentage of your audience you want to prevent from receiving a message. We’ll still generate deliveriesThe instance of a message sent to a person. When you set up a message, you determine an audience for your message. Each individual “send”—the version of a message sent to a single member of your audience—is a delivery. for your holdout group, but we send them to an internal message trap rather than sending them to your audience.

This lets us calculate conversions, so you can compare [conversionsA campaign goal that you want your audience to achieve. You measure this goal as an attribute change or an event that occurs after a person receives a message within a campaign workflow.](/journeys/campaign-conversions/) for people who receive a message to conversions for people who don’t, and determine the true utility of your message.

## Create a holdout test[](#create-a-holdout-test)

It tends to be easier to set up an A/B test where one variant is the holdout. But, if you want to test multiple variants of a message, you can also [perform holdout tests with a random cohort branch](#random-cohort-holdout).

Either variation in your A/B test—email A or B—can be the holdout. But, for this example, we’re treating variation B as the holdout.

1.  In your campaign’s **Workflow**, select the email you want to convert to a holdout test.
    
2.  Click **Turn into A/B test**.
    
    [![image.png](https://docs.customer.io/images/a-b-test-email.png)](#3fd1d859a7169f7f93b2302e1427568d-lightbox)
    
3.  Set the amount of traffic that you want to send to your message (variation A) and holdout group (variation B).
    
    [![Set the percentages of people who will receive a message and who won't](https://docs.customer.io/images/test-percentages.png)](#528948e117ceddd14282df55208b29fd-lightbox)
    
4.  Click **Edit Variation B**, and select **Make this message a holdout test**. This converts the “message” to a holdout, ensuring that it’s sent internally to a message trap; it is not sent or visible to your audience.
    
     Check your sending behavior
    
    The *Sending Behavior* has to be set to *Queue Draft* or *Send Automatically* for your holdout test to work because the holdout test “sends messages” to the message trap. If you use the *Don’t Send* setting, your holdout test won’t work.
    
    [![Hold message b out of the test](https://docs.customer.io/images/holdout-checkbox-1.png)](#d4758f86a2b18085bea3df9926ad7ba9-lightbox)
    
5.  Now, when you start your campaign, some people will receive *Message A*, and some people won’t get a message at all.
    

## Random cohort holdout testing[](#random-cohort-holdout)

If you’ve set up your own holdout test by creating a “black hole” address, or you want to test more than the two variants supported by an A/B test, you can use a random cohort branch to perform a holdout test.

1.  Drag a **Random Cohort Branch** into your campaign or broadcast workflow.
    
2.  Click the *Random Cohort Branch* action, set the percentage of people who will flow down each path, and click **Save**. Click **Add Path** if you want more than two branches in your random cohort.
    
    [![image.png](https://docs.customer.io/images/random-cohort-1.png)](#7115cac5ff2f49781124b533c6d043d4-lightbox)
    
3.  Add messages to each branch in the test, and set up your message content for the “live” message. You don’t need to write content for the “holdout” variant, because it won’t be sent to your audience.
    
4.  Click the message that you want to represent your holdout, and select **Make this message a holdout test**. This converts the “message” to a holdout, ensuring that it’s sent internally to a message trap; it is not sent or visible to your audience.
    
    [![Hold message b out of the random cohort](https://docs.customer.io/images/holdout-checkbox-random-cohort-1.png)](#0afe427d452a4051f388eae37a549637-lightbox)
    

Now, when you start your campaign, a percentage of people will flow through your random cohort branch to the holdout variant. These people won’t receive a message.

## Check holdout test performance[](#check-holdout-test-performance)

To check your results, go to your campaign and click the **A/B Test** tab. Here, you can select the “winner” of the test. The winner remains in your campaign, and the other message is removed. If your winner is the holdout, the message action is removed from your workflow entirely.

[![Determine the utility of your message in the A/B test tab](https://docs.customer.io/images/holdout-performance.png)](#0a216267a975cc991dd76b341cad1169-lightbox)

You can determine the winner using the *Chance To Beat Original* (CTBO) metric, in three different statistics: open rate, tracked link clicks, and conversion rate. You can learn more about how we calculate CTBO [here](/journeys/understanding-ab-results).

When you use holdout tests, you should [set conversion criteria](/journeys/campaign-conversions/) and use the conversion metric to determine the true utility of your message. We don’t send a “holdout” message, so it’ll never be opened or clicked; the message variant that you send will always win those metrics. But in holdout tests, people in the holdout group register conversions when they perform the goal action within the conversion time window—even though they didn’t receive any message. This helps you measure the true impact of your messages by comparing conversion rates between those who received messages and those who didn’t.

If there isn’t a clear winner in your test result categories, we’ll show a “Not significant, need more data” message. You might need to let your test run longer to gather more results. If your test has already run for a while, and you’re confident that enough people have gone through the test, there might not be a statistically significant difference between your message and the holdout.

## Holdout message status and metrics[](#holdout-message-metrics)

Messages that are held out are “sent” and “delivered” to a message trap in Customer.io. We do this purely to help you visualize the performance differences between your real messages and your “held out” messages. But these messages never leave Customer.io.

Because holdout messages never leave Customer.io, they don’t affect your email domain’s deliverability ratings.

When you check the **Sent** tab from your campaign, holdout messages show their status as **Holdout**, so you can differentiate between your real messages from your holdout tests.

[![messages in a holdout test show their status as holdout](https://docs.customer.io/images/holdout-test-metrics.png)](#76f44074c73074ba9607228b86c4b117-lightbox)