A/B Testing

A/B testing is a method of comparing two versions of something, such as a web page, email or ad, by showing each version to a random group of people and measuring which one performs better. Version A is usually the original, and version B is the change you want to test. It matters because it lets you improve results with evidence, rather than guessing what your audience prefers.

Key Takeaways

• It is also called split testing or bucket testing.
• Traffic is split at random, so the only real difference between the groups is the change you made.
• You need a clear goal, such as sign-ups or sales, before you start.
• Statistical significance tells you whether a difference is likely to be real or just chance.
• Most ad and email platforms have built-in tools. Google Ads experiments and Mailchimp A/B tests are two examples.

What Is A/B Testing?

Optimizely, an experimentation software company, defines it as “a methodology for comparing two versions of a webpage or app against each other to determine which one performs better.” It adds that the method is “also known as split testing or bucket testing.”

Here is a simple example. An online shop wants to know whether a green or orange “Buy now” button gets more sales. Half of its visitors see green and half see orange, chosen at random. After enough visits, the shop compares the sales from each group. The colour with more sales wins.

The same idea works almost anywhere in marketing. For instance, you can test subject lines, landing page headlines, ad images or offers.

How A/B Testing Works

Optimizely describes the core steps. First, you create two versions: “the original (control or A) and a modified version (variation or B).” Next, you split your traffic at random between them. Finally, you measure the results and use “statistical analysis to determine which variation achieves better results for your conversion goals.”

Why Random Splits Matter

Randomness is what makes a test fair. If you showed version A on Monday and version B on Tuesday, a busy day could skew the result. By contrast, a random split means both groups see the same mix of days, devices and people. So when one version wins, you can trust that the change caused it.

Statistical Significance

Small tests can mislead you. If 10 people see each version, one extra sale can look like a huge win. That’s why testing tools check statistical significance, which estimates how likely it is that the difference is real. Optimizely lists “check statistical significance” as a key step before you act on results.

Testing in Ad and Email Tools

Many platforms build this in. In Google Ads, an experiment “shares your original campaign’s traffic (and budget),” and you choose what percentage of the budget goes to the test. For Search campaigns, you can split by cookies or by searches. Meanwhile, Mailchimp says you can test “1 of 4 variables: subject line, From name, content, or send time,” with up to three variations.

What to Test First

If you are new to testing, start where the most people see your work. For example, a home page headline, an email subject line or the main image in an ad all reach a large share of your audience. As a result, even a small improvement there adds up quickly. Pages and emails that few people see take much longer to give a clear answer, so save them for later, once your busiest pages are working well.

A/B Testing Examples

Here are three made-up examples of tests a marketing team might run:

• Email subject line: A charity tests “Your gift doubles today” against “Help us reach our goal.” The winning line, chosen by click rate, goes to the rest of the list.
• Landing page: A course provider tests a long page against a short one. As a result, it learns that the long page brings more enquiries from serious buyers.
• Google Ads bidding: A shop runs an experiment that sends half the budget to a new bid strategy. After a few weeks, it compares cost per sale before switching fully.

A/B Testing Tools Compared

This table compares how common tools handle tests:

How to Run an A/B Test

Follow these steps to run a clean test:

1. Pick one goal. Choose a single main metric, such as sign-ups, sales or click rate.
2. Write a hypothesis. For example, “A shorter form will increase sign-ups because it takes less effort.”
3. Change one thing. If you change the headline and the image at once, you won’t know which one made the difference.
4. Split traffic at random. Use your platform’s built-in tool, so the split is fair.
5. Wait for enough data. Let the test run until your tool shows a reliable result.
6. Check significance and side effects. Look at your main metric, then check that other metrics did not suffer.
7. Record and roll out. After that, apply the winner and write down what you learned.

Best Practices for A/B Testing

• Test big ideas first. A new offer or headline usually moves results more than a small colour change.
• Keep a test log. Record each hypothesis, result and decision, so the team learns over time.
• Run full weeks where you can. This smooths out differences between weekdays and weekends.
• Accept losing tests. Optimizely notes that “not every test will be a winner,” and a clear loss still teaches you something.
• Test trust signals too. Nielsen Norman Group suggests A/B testing comments, reviews, likes and testimonials to see which ones increase conversions.
• Measure what matters. A higher click rate is only good if it leads to more conversions.

Common A/B Testing Mistakes

Several errors make tests unreliable. First, people stop a test early as soon as one version looks ahead. Second, they test several changes at once and cannot tell what worked. Third, they run tests on tiny audiences, so random noise decides the winner. Finally, they never write down results, so the same ideas get tested again and again. Each of these is easy to avoid with a simple plan before you start.

Where You’ll Use A/B Testing at Work

Testing skills appear across marketing job listings. For example, recent remote roles ask for:

• “Solid A/B testing basics: sample size, a single primary metric, a minimum effect worth acting on” (Website and Lifecycle Marketing Lead).
• The ability to “Run A/B tests across audiences, messaging, subject lines, offers, landing pages, calls to action, creative, and channels to improve conversion and efficiency” (Demand Generation Manager).
• “Strong hands-on experience designing, running, and interpreting A/B tests, including experimentation methodology and statistical principles” (Senior Growth Manager, Web Experience and CRO).

So whether you become a Growth Marketer, Digital Marketing Manager or Freelance Digital Marketer, you will plan tests, read the results and explain what to change next.

Learn It in the Full Stack Programme

You can practise testing across ads, emails, social posts and web pages in the Full Stack programme, which covers all four tracks and all ten courses. Along the way, you complete one portfolio project from each track, so you can show tests and results to employers. You also join a U.S. Project Internship from day one, with unlimited 1-on-1 mentoring.

For more ideas on what to test, read our guide to split testing.

Frequently Asked Questions

What is A/B testing in simple terms?

It is showing two versions of something to two random groups and seeing which one gets better results. You then use the winner.

Is A/B testing the same as split testing?

Yes. The terms mean the same thing, and some tools also call it bucket testing.

How long should an A/B test run?

Long enough to reach a reliable result in your testing tool. For many sites, that means at least one full week to cover different days.

What can I A/B test in email?

In Mailchimp, you can test the subject line, From name, content or send time, with up to three variations.

What is the difference between A/B testing and multivariate testing?

A/B testing compares two versions with one main change. Multivariate testing changes several elements at once to see which combination works best, so it needs much more traffic.

Sources: Optimizely: A/B testing; Google Ads Help: Set up a custom experiment; Mailchimp: About A/B testing campaigns; Nielsen Norman Group: Social proof in the user experience; remote job listings on Himalayas.