On this page
Create A/B test campaigns
Send two versions to a sample, let MailMachine pick the better one by clicks, and send the winner to the rest.
A/B testing is an advanced feature available on selected plans. If you don't see A/B Tests in your account, it may not be included on your current plan - get in touch and we'll let you know your options.
An A/B test (or split test) sends two versions of a campaign to a sample of your list, sees which gets more clicks, and then sends the winner to everyone else - so MailMachine picks the better email for you. It's one of the most reliable ways to improve your results over time. This guide covers setting one up.
A/B testing needs a big enough sample to find a clear winner, since the decision is based on clicks. On a very small list, or a time-sensitive send, it may not reach a meaningful result in time - work out how long it'll take to gather enough clicks before you rely on it.
Set up an A/B test
- Click the Campaigns tab and select A/B Tests.
- Click the + to start a new test.
- Name it, choose a list and segment, and click Continue.
- Select the test block to open its distribution settings on the left:
- By default, A and B are split 50/50.
- Set the test distribution size and the winner size. For example, with a 40% test size and a 60% winner size, 40% of contacts receive the A/B sample and the better-performing version then goes to the remaining 60%.
- Set the winner criteria: when the test ends, and a fallback email to send if no clear winner emerges (when there isn't enough of a difference to be statistically meaningful).
- Select the Email A block, click Edit, and build it.
- To reuse A's content as a starting point for B, click Copy Email A Content to B, then vary it.
- Click Save and Exit, and do the same for Email B.
- When you're happy, click Start Test at the top right.
Your test runs, and you'll find it in your A/B test drafts to analyse. You can rename, delete or duplicate it from the arrow menu.
Troubleshooting
A/B testing isn't available - check it's included on your plan, and that your list is big enough to be meaningful.
Results are inconclusive - increase the sample size (around 1,000+ contacts is a sensible minimum), extend the test duration, and make sure your two versions are different enough to tell apart.
Winner-selection issues - review your criteria, always set a fallback email, and allow several hours for engagement to accumulate.
Distribution problems - make sure your test and winner percentages add up sensibly, and that your segments don't overlap.
For pre-send checks, see Testing your campaign. Stuck? Open a support ticket.