Experiments

Learn how to use experiments to test different versions of your emails.

Experiments are a way to run split tests within loops. This lets you test different versions of your emails, allowing you to see which version performs best.

You can test things like subject lines, preheaders, and content, and specify the percentage of contacts who should be sent down each variant branch.

Creating an experiment

To add an experiment, add a new node by clicking the + button in the loop builder and select Experiment. You can add an experiment to any loop.

You can add multiple experiments to a loop—even within other experiments if you want. Each experiment can contain any number of variants.

Each variant can contain emails, timers and audience filters.

Variants and controls

Experiments are made up of variant and control branches.

• Variant branches contain the email(s) you are testing, which will be sent to a specific percentage of contacts that you define.
• The Control branch contains the email(s) that will be sent to all contacts not included in the experiment, and should be used as the baseline for the experiment. A control is optional; you can create an experiment without a control branch.

An experiment node must be followed by at least one variant or control branch, otherwise you cannot start the loop.
Likewise, you must add at least one email within each control or variant branch before starting the loop.

Sample size

You can specify the percentage of contacts who will be sent to control and variant branches by dragging the slider in Experiment nodes.

Every variant you add will receive an equal percentage of contacts within the experiment sample. For example, if you set the sample size to 60% and add three variants and a control, 20% of contacts will be sent to each variant branch (60% / 3) and 40% of contacts will be sent to the control branch.

You can change a variant to be the control by clicking on the variant node and toggling the Use as control option.

If you do not add a control branch, only your selected sample size will be sent through the experiment node. For example, if you have three variants and no control and select 60% for your sample size, 40% of contacts entering the loop will exit at the experiment node and not receive any emails.

Editing experiments

Once you have started your loop you can edit the number of contacts passing through each variant and control by first pausing the loop from the top right of the loop builder.

Once you have paused the loop, you can change the sample size, edit your variants and controls, and update any content.

Click Resume to start the loop again.

When paused, email sending is stopped and contacts are queued behind their respective nodes as expected.
All contacts that were scheduled to receive an email during the pause will receive it once you resume the Loop.

Closing an experiment

After some time testing, you may want to close experiment branches, to send all future contacts down the your best-performing branch.

To do this while retaining the metrics for all branches, first change your best-performing branch to a control branch (click its Use as control toggle; see image above).

Then set the Experiment node sample size to 0%. This will change the Control size to 100%, sending all contacts down the new control branch.

Results

Once your experiment is running and contacts have flowed through the loop, you can see the results by visiting the Metrics tab.

Here you can compare your variants and view metrics for sends, opens and clicks.