Email A/B testing: what to test and how to read results

An email A/B test sends two versions to randomly split groups. Pick the outcome and the window before you send, then decide once both versions reach their planned size.

How an email A/B test works

Split one audience at random, send each half a different version, and compare one outcome picked in advance.

3,600 new trial users
A1,800 people
A
Your Acme workspace is readyAcme
Acme

Add your first contacts

Hi First Name, your workspace is set up. Next, bring in your contacts.

Finish setup
B1,800 people
A
Your Acme workspace is readyAcme
Acme

Add your first contacts

Hi First Name, your workspace is set up. Next, bring in your contacts.

Import your contacts
Activated within 7 dayspicked before the first send
A
10.0% 180 of 1,800
B
13.0% 234 of 1,800
B: 13% vs 10%, at the full 1,800 per version.Keep B.
Example numbers.

What to test

Change one thing to learn what caused a difference. Change several and the result applies only to the whole version.

MMaya at Acme9:00 AM
Your Acme workspace is ready
Two quick steps before your first send
Acme

Add your first contacts

Hi First Name, your workspace is set up. Next, bring in your contacts.

Import your contacts
Each label is something you can test on its own.

Test send time across several sends, since one day can mislead. For the inbox fields, see sender name and preview text.

If your goal is activation or upgrades, let that outcome decide. Subject lines are quick to test, but privacy features distort opens. Campaign goals in Loops count activations or upgrades after a send.

How to run a test that tells you something

1
Before you buildChange one thing

Keep the sender, subject, and body the same. Change the button and the subject together, and a lift can’t tell you which one helped.

Version AVersion BSenderAcmeAcmeSubjectYour Acme workspace is readyYour Acme workspace is readyButtonFinish setupImport your contactsBodySame copySame copy
2
Before you sendPick the metric first

Choose activation, upgrades, or another outcome you care about. Opens and clicks help explain a result, so don’t switch to them after you see the numbers.

Test planwritten before the first send
Decides the testActivated within 7 days
Explains itOpens and clicks
Keep B only ifit’s ahead once both reach their planned size
3
Before you sendPlan the sample size

Start from your baseline rate and the smallest lift worth acting on. Rare outcomes like paid upgrades need large groups, even on a big list.

Baseline activation10%
Smallest lift worth acting on+3 pts
Confidence and power95% and 80%
People per version≈ 1,8003,600 in total
4
While it runsFix the end date

Everyone needs the full outcome window. An early lead can change as more people finish it, so don’t stop because it looks good.

early leadRead the result
Day 0Day 7Day 14Day 21
New users join the testThe last to join get their 7 days
5
After the windowCheck you hit the plan

Compare the versions only once both reach their planned size. If the test stopped short, keep the current version. Then check both were delivered alike before you credit the copy.

Example numbers. Sample size from a standard two-proportion calculation.

How to read the result

Only a test that reaches its planned size can be trusted.

Ran to planB wins. Keep it.
A
10%
B
13%
People reached, per version1,800 of 1,800
Stopped earlyToo early to call. Keep testing.
A
10%
B
13%
People reached, per version400 of 1,800
Example numbers, using the 1,800-per-version plan from step 3.

At 400 people per version, a 3-point gap can come from chance alone.

How to A/B test in Loops

A/B tests in Loops run inside workflows with an Experiment node. The sample splits evenly across the variants, and everyone else goes to the Control branch.

Experiment
50% → 1 variant
50% → control
Variant
50% distribution
Send email
Your Acme workspace is ready
Control
50% distribution
Send email
Your Acme workspace is ready
Each branch needs at least one email before you can start the workflow.

Metrics compares sends, opens, and clicks per branch. For activation or upgrades, record who got which version with the loop.email.sent webhook, which carries the contact’s userId and the email’s emailMessageId.

To close the test, turn on Use as control for the winner and set the sample to 0%. Each branch keeps its metrics (Experiment node docs).

Set it up with an AI agent

Connect Claude, ChatGPT, Claude Code, or Codex to the Loops MCP server, then paste a prompt:

Build the testCreate a draft workflow in Loops for new contacts with an Experiment node at 50%: our current setup email as the Control and a copy of it as the variant.
Write version BIn the variant email, change only the button to Import your contacts and keep everything else the same. Send me a preview of both versions.
Check deliveryPull sends, opens, clicks, bounces, and spam reports for both emails in the experiment, and flag any delivery gap before I compare activation.

The agent builds a draft with the Workflows API, which is in alpha. It can’t start workflows, so you review the draft and start it.

Estimate incremental lift with a holdout

To measure what a whole lifecycle sequence adds, randomly hold some eligible users out of it. Keep required transactional email running for both groups.

Assign each user once in your app before enrollment, store the group, and sync it to Loops as a contact property. Filter the workflow to the treatment group. For account-level outcomes, assign whole accounts together.

Count outcomes for everyone assigned over the same window from enrollment, including people who never open and treatment users whose email bounced.

Activated within 7 daysToo early to call. Keep it running.
Got the sequence120 of 1,000
12%
Held out100 of 1,000
10%
People reached, per group1,000 of 3,800
Example numbers. Spotting a 2-point lift reliably takes about 3,800 people per group.

At 1,000 per group, a 2-point gap can still come from chance, so keep the holdout running until both groups reach their planned size. See how to define conversion rates and attribution windows.

Frequently asked questions

What is a good sample size for an email A/B test?

It depends on how often the outcome happens today and the smallest change worth acting on. Work out the number before the test starts. A few hundred recipients doesn’t guarantee a useful answer.

How long should an email A/B test run?

Use a planned enrollment period and let every participant complete the outcome window. A seven-day activation test needs seven days of observation after the final enrollment. Avoid stopping early because the result looks favorable.

What should I A/B test first?

Start with a change likely to affect your primary outcome. For activation, test the prompt or next action. For upgrades, test the offer or explanation of value. Subject lines are one option, and opens are a diagnostic metric.

Can I A/B test in Loops?

Yes. Use an Experiment node inside a workflow to compare email variants. Its metrics show sends, opens, and clicks. Track product conversions with recorded assignments in your analytics, and use a no-email holdout when estimating incremental lift.

What's next

More on marketing email.

See all guides