Email A/B testing: what to test and how to read results
An email A/B test sends two versions to randomly split groups. Pick the outcome and the window before you send, then decide once both versions reach their planned size.
Split one audience at random, send each half a different version, and compare one outcome picked in advance.
Add your first contacts
Hi First Name, your workspace is set up. Next, bring in your contacts.
Finish setupAdd your first contacts
Hi First Name, your workspace is set up. Next, bring in your contacts.
Import your contactsChange one thing to learn what caused a difference. Change several and the result applies only to the whole version.
Add your first contacts
Hi First Name, your workspace is set up. Next, bring in your contacts.
Test send time across several sends, since one day can mislead. For the inbox fields, see sender name and preview text.
If your goal is activation or upgrades, let that outcome decide. Subject lines are quick to test, but privacy features distort opens. Campaign goals in Loops count activations or upgrades after a send.
Before you test, check email deliverability best practices and why emails go to spam.
Only a test that reaches its planned size can be trusted.
At 400 people per version, a 3-point gap can come from chance alone.
A/B tests in Loops run inside workflows with an Experiment node. The sample splits evenly across the variants, and everyone else goes to the Control branch.
50% → control
Metrics compares sends, opens, and clicks per branch. For activation or upgrades, record who got which version with the loop.email.sent webhook, which carries the contact’s userId and the email’s emailMessageId.
To close the test, turn on Use as control for the winner and set the sample to 0%. Each branch keeps its metrics (Experiment node docs).
Connect Claude, ChatGPT, Claude Code, or Codex to the Loops MCP server, then paste a prompt:
The agent builds a draft with the Workflows API, which is in alpha. It can’t start workflows, so you review the draft and start it.
To measure what a whole lifecycle sequence adds, randomly hold some eligible users out of it. Keep required transactional email running for both groups.
Assign each user once in your app before enrollment, store the group, and sync it to Loops as a contact property. Filter the workflow to the treatment group. For account-level outcomes, assign whole accounts together.
Count outcomes for everyone assigned over the same window from enrollment, including people who never open and treatment users whose email bounced.
At 1,000 per group, a 2-point gap can still come from chance, so keep the holdout running until both groups reach their planned size. See how to define conversion rates and attribution windows.
What is a good sample size for an email A/B test?
It depends on how often the outcome happens today and the smallest change worth acting on. Work out the number before the test starts. A few hundred recipients doesn’t guarantee a useful answer.
How long should an email A/B test run?
Use a planned enrollment period and let every participant complete the outcome window. A seven-day activation test needs seven days of observation after the final enrollment. Avoid stopping early because the result looks favorable.
What should I A/B test first?
Start with a change likely to affect your primary outcome. For activation, test the prompt or next action. For upgrades, test the offer or explanation of value. Subject lines are one option, and opens are a diagnostic metric.
Can I A/B test in Loops?
Yes. Use an Experiment node inside a workflow to compare email variants. Its metrics show sends, opens, and clicks. Track product conversions with recorded assignments in your analytics, and use a no-email holdout when estimating incremental lift.