Why is Amazon so successful? A big part of the answer is A/B Testing. Unknown to most users, we are subject to thousands of A/B tests every week. It tells Amazon how to layout a page, which products to offer you, and even the font size on the buy buttons.
A/B testing is ubiquitous on the internet and it powers millions of marketing decisions a day. Yet the idea is shockingly simple. What has changed is the technology that allows marketers to carry out their A/B tests on statistically significant samples quickly and easily. That’s what the internet and web technologies have given us.
So what is this age-old idea, and what can it do for you?
What is A/B Testing?
Let’s take a simple example. You have a website with pages that sell your books. You have an ‘Add to Basket’ button and you wonder: ‘what if I changed the design of the button?’ Would a change in design affect the percentage of people visiting your sales page who click the button?
So, what you do is set up an A/B test. You create two versions of your webpage and use a simple software tool to serve one of the two versions randomly to everyone who selects that page. The software then tracks how many people click the button.
In A/B testing we usually have three conventions:
- One version is always the current version. Taking our language from the world of scientific experiments, we call this the control.
- Test one thing at a time. If you tried to test the button design and the page background colour, and you got an improvement in click-through rates, which one caused the improvement? And did the other help or reduce that improvement?
- A test is only conclusive if the change in behaviours for the B state is ‘statistically significant’. That means that there is enough difference in behaviour to be confident that it is due to the change you made, rather than just who happened to visit.
The Power of the Internet
It’s the third of these conventions that really harnesses the power of the web. It’s the sheer number of people out there. If you have a popular site like Amazon, you can get thousands of people seeing the variations in a short time. This means you can learn a lot, quickly.
Crudely, a statistically significant change is 1/√n where n is the sample size. So, if;
- 100 people visit your web page in a month, you need more than 10% difference to be confident that it’s due to the change you made
- 10,000 people visit your web page in a month, you only need more than 1% difference to be confident that it’s due to the change you made
What Next?
There are three scenarios that can flow from your first round of A/B testing. Two have much the same implication:
- No significant variation in behaviour.
Define a new A/B test. Try making a bigger change. - The variation is adverse and you see less of the behaviour you want.
Stop the test, keep your control, and try a new test. - You get a significant improvement in the numbers.
Stop the test, keep the new version as your next control, and try a new test.
Multiple A/B Testing
Can you test more than one thing at a time?
Yes, you can. But, if you make multiple changes to create your B state, this is usually referred to as Split Testing.
However, if your software allows, you can test multiple changes in one go, by having an A state as control, and then B, C, D… states with different variations. In our example, each could have a different button design.
Big organisations are constantly A/B testing and Split testing their marketing to improve conversion rates.
What Can You Split Test?
Pretty much anything. It depends largely on:
- The software tools you have at your disposal
- The imagination of your team
Common examples of A/B testing include:
- Key messages like which benefit to lead with
- Sales and marketing copy – the text on web pages, emails, social media adverts, and their real-world equivalents
- Design elements, like font, layout, colour, size
- Audience targeting in adverts
- Automation sequences
- What promotions you offer
- Pricing (to compare profitability)
Why is A/B Testing Relevant to Managers
Okay, you may not be a marketer. So, what use is all of this? It’s not the technology, nor even the examples. Here, it’s the Big Idea behind A/B testing itself.
As a manager, a part of your job is to make improvements in productivity, efficiency, and effectiveness. So, what A/B tests can you design in your day-to-day work and in the way your team does its job, to look for improvements?
What is Your experience of A/B Testing?
We’d love to hear your experiences, ideas, and questions. Please leave them in the comments below.
To learn more…
If marketing interests you, we have a lot of articles on marketing – especially in or Big Ideas and Management Thinkers series.
You may also like The Marketing Pocketbook. It’s full of tips, techniques, and tools for all those involved in the marketing process – research, strategy, planning and tactics.
