Email A/B testing: what to test and how to read results
An email A/B test compares versions sent to randomly assigned groups. Choose the outcome and observation window before sending, then use the result and its uncertainty to decide whether to keep the change.
What to test
Change one thing when you want to learn what caused a difference. Testing complete alternatives is also valid, but the result applies to the whole version. Common options include:
Subject line. Compare a direct description with a different way of introducing the same message.
Sender name. Compare your brand name with a recognizable person, such as Chris at Loops.
Preview text. Test the snippet that appears beside the subject in the inbox.
Send time. Useful, but test it on its own and read results across several sends, since a single day can mislead.
Call to action. Button wording, one CTA versus several, and placement near the top versus the bottom.
Content and layout. Long form versus short, image heavy versus text, single column versus multi.
Choose the test around your goal. Testing subject lines can help, but for activation or upgrades choose a downstream outcome as the primary metric. Open tracking is affected by privacy features.
How to run a test that tells you something
Change one variable. If you swap the subject and the CTA at once, a lift tells you nothing about which one caused it.
Pick the metric before you send. Use activation, paid upgrades, or another intended outcome as the primary measure. Opens and clicks can help diagnose the result, but do not change the winning metric after seeing the numbers.
Plan the sample size. Use your baseline conversion rate, the smallest improvement worth acting on, and a chosen significance level and statistical power. There is no universal minimum number of recipients.
Set the end date and outcome window. For a seven-day activation metric, every included participant needs seven days of observation. Do not stop a fixed-horizon test because an early result looks favorable.
Check uncertainty. Report a confidence interval alongside the difference. An inconclusive test does not prove that the versions perform equally.
Mistakes that void your results
Calling it early. An early lead can change as more participants complete the outcome window.
Too few outcomes. A large recipient count can still produce little evidence when paid upgrades are rare. Judge the result against the planned sample and its uncertainty.
Overinterpreting a version test. If subject and content both change, the test compares the complete versions. It does not isolate which edit helped.
Measuring the wrong metric. If your goal is activation, an increase in opens alone does not answer whether the change helped.
Ignoring deliverability. Check whether differences in delivery contributed to the result before attributing it to the copy.
Check authentication and list hygiene before testing. See email deliverability best practices and why emails go to spam.
Estimate incremental lift with a holdout
A version test compares two emails. To estimate the effect of sending a lifecycle sequence, randomly assign eligible users to receive it or to a holdout that does not receive that sequence. Keep required transactional messages running for both groups.
Make the assignment once in your app before enrollment and store the experiment ID, group, and enrollment timestamp. Sync a group property to Loops and filter the tested workflow to the treatment group. Keep holdouts out of other copies of that sequence. For an account-level outcome, assign the whole account together to avoid teammates receiving different treatments.
Count outcomes from every assigned eligible user or account in each group, including people who never open or click and treatment recipients whose email bounces. Start the same outcome window at enrollment for both groups. Compare concurrent groups with the same eligibility rules and leave other messaging unchanged.
Illustrative result: 120 of 1,000 treatment users activate within seven days, compared with 100 of 1,000 holdout users. Rates are 12% and 10%. The estimated lift is 2 percentage points, or 20% relative to the holdout. That corresponds to an estimated 20 additional activations per 1,000 assigned treatment users.
The point estimate alone is insufficient. In this example, a simple two-proportion normal approximation gives a 95% interval of roughly −0.7 to +4.7 percentage points, which includes zero. Report it as inconclusive. Choose the sample size and analysis method before starting, check that the actual allocation matches the plan, and let the final cohort finish its outcome window.
How to A/B test in Loops
In Loops, A/B testing lives inside Workflows, using an Experiment node.
Build a workflow and add an Experiment node where you want the split.
Create the variants on each branch, for example two different emails or two subject lines.
Loops splits incoming contacts between the branches so each variant reaches a comparable group.
Compare sends, opens, and clicks in the Experiment metrics. Measure activation or paid upgrades in your product analytics with recorded experiment assignments. Select a winning version only after the planned analysis.
Each variant and the optional Control branch must contain an email. The Control branch is an email baseline. With no Control branch and a sample below 100%, the remaining contacts exit without email. Record holdout membership and outcomes separately if you use that configuration for a lift study.
Frequently asked questions
What is a good sample size for an email A/B test?
How long should an email A/B test run?
What should I A/B test first?
Can I A/B test in Loops?