Simple A/B testing explained – without fooling yourself

A/B testing sounds like something reserved for big companies with their own data teams. It isn't. At its core it's a simple idea – but one that's surprisingly easy to mess up.

The idea is this: you show two versions of a page to two groups of visitors, and see which one converts better.

What an A/B test actually is

You make two variants. A is the one you have today. B is a version with one change – a new headline, a different button label, a shorter form.

Half the visitors see A, half see B, split at random. After a while you compare the conversion rate. If B clearly won, you keep it. If neither won, you've at least learned something.

The nice part is that you're not guessing. You let the actual visitors vote with their actions.

Test one thing at a time

The most common mistake is changing several things at once. If you swap the headline, the button and the image in variant B, and B wins, you don't know which one made the difference.

Stick to one change per test. It's slower, but it's the only way you learn anything. If you want to test five ideas, run five tests – not one test with five changes.

You need enough traffic

This is the thing people most often overlook. A test on a page with twenty visitors a week gives you nothing you can trust. The difference you see is just chance.

The rule of thumb: the less traffic you have, the longer the test must run, and the bigger the difference has to be before it means anything. If you have very little traffic, A/B testing is simply the wrong tool – you'll learn more from talking to customers than from waiting for numbers that never get clear.

Let the test run long enough

Never end a test because B looks set to win after two days. Early numbers swing a lot, and it's easy to see a "winner" that disappears the following week.

Two things should govern when you stop:

• Time: let the test run for at least a full week, ideally two, so you catch differences between weekdays and weekends.
• Volume: wait until each variant has had enough visitors and enough conversions that the difference isn't just luck.

If you stop too early, you're measuring noise and calling it insight.

Don't stop the moment you've seen the numbers

A subtle trap: you check the result every day and stop the test on exactly the day B happens to be ahead. That's a great way to fool yourself.

Decide in advance how long the test will run, and stick to it. Let the test finish before you draw conclusions.

What to test first

Start with the things that affect many visitors and are easy to change:

• The headline on your most important page
• The wording on the main button
• The number of fields in the contact form
• What the offer actually is, phrased two ways

Small, cheap changes that reach many people give you more to learn from than big rebuilds that are hard to interpret.

When you don't need a test

Sometimes you don't need an A/B test to know the answer. If the page loads in eight seconds, it isn't a test that's missing – it's a fix. If the form is missing a submit button, don't test it, fix it.

A/B testing is for the cases where two reasonable options are genuinely undecided. It's not a substitute for correcting obvious problems.

Used well, A/B testing is a way to let customers tell you what works, instead of arguing about it internally. If you'd like to get started with a simple, honest test on your site, we're happy to help you set one up.