A/B Testing & Experimentation

The £15 headline change that increased revenue by £340,000

(The following scenario is illustrative — based on a common A/B testing pattern. Specific companies, dates, and exact figures are constructed for teaching purposes.)

In 2019, a UK insurance comparison site was reviewing their landing page. The page converted well — 4.1% — but the growth team had a hypothesis.

Their current headline: "Compare insurance quotes from 40+ providers." Their hypothesis: leads were comparison-shopping, not outcome-seeking. The headline talked about the tool, not the benefit.

New headline: "Stop overpaying for insurance. Compare in 2 minutes."

They ran a proper A/B test: 50/50 traffic split, 14-day test, 3,200 sessions per variant.

Result: the new headline converted at 6.3% vs 4.1% — a 53.7% improvement that was statistically significant.

At 800 leads/day, that improvement was worth an extra 139 leads every day. At a close rate of 22% and average policy value of £170, that's £5,196 additional revenue daily — or £1.9M per year (illustrative — actual revenue impact depends on this company's actual close rate and policy value).

The headline cost £15 to change (half an hour's copywriter time). The A/B test cost four weeks of patience.

What A/B testing is — and isn't

What it is: A controlled experiment where two versions of something (A and B) are shown to randomly split audiences simultaneously. The version that performs better wins.

What it isn't: Showing version A to last week's audience and version B to this week's. Showing version A on mobile and version B on desktop. Picking the winner after one day because one "looks" better.

The fundamental requirement: Both versions must run simultaneously, on randomly split, comparable audiences, for long enough to produce statistically valid results.

The statistical concepts you can't ignore

Why you can't just pick the winner after 100 visitors:

Imagine flipping a fair coin 10 times. You might get 7 heads — does that mean the coin is biased? No — with only 10 flips, 7 heads can happen by random chance quite often. With 1,000 flips, 700 heads would be extraordinary evidence of bias.

A/B tests work the same way. With too few visitors, apparent differences are often random noise, not real effects. Running a test too early and calling a winner is one of the most common and expensive A/B testing mistakes.

Statistical significance: The probability that the difference you observed isn't due to chance. The standard threshold: 95% confidence (meaning there's only a 5% chance the observed difference is random).

Statistical power: The ability of the test to detect a real effect if it exists. Insufficient traffic means a real improvement might not show up as statistically significant.

Minimum Detectable Effect (MDE): The smallest improvement you want to be able to detect. If you're looking for a 10% improvement in a 4% conversion rate (i.e., from 4% to 4.4%), you need more traffic than if you're looking for a 50% improvement.

Sample size calculators (use one): Before starting any A/B test, calculate required sample size. Inputs: current conversion rate, minimum detectable effect, desired confidence level. The calculator tells you how many visitors per variant you need before declaring a winner.

A rough rule of thumb: For a 4% conversion rate and a desire to detect a 20% relative improvement (e.g., from 4% to 4.8%) at 95% confidence, standard calculators return roughly 4,500–5,200 visitors per variant (for a two-tailed test at 80% power). Smaller effects require substantially more visitors — this is why defining your minimum detectable effect matters before running the test. Always use a sample size calculator for the specifics.

What to test (and in what order)

Not all tests are created equal. High-impact elements produce bigger improvements and are worth testing first:

Highest impact (test first):

• Headline: The most-read element on any page. A 10–30% conversion impact is not unusual.
• Primary CTA: Button text, colour, placement, and the offer itself
• Value proposition: The core promise of the page — what problem you solve, for whom
• Social proof: Presence/absence of testimonials, number of reviews, specific proof claims
• Pricing structure: Free trial vs. freemium, pricing display, anchoring

Medium impact (test second):

• Body copy length and structure
• Image vs. video as hero media
• Form length (fewer fields typically increases submissions but reduces quality)
• Trust signals (logos, certifications, guarantees)

Lower impact (test last):

• Button colour (does matter, but less than the above)
• Font size and styling
• Background colour
• Footer content

The one-variable rule: Test one change at a time. If you change the headline AND the button colour AND the image, you can't know which change drove the improvement (or the decline).

There Are No Dumb Questions

"How long should I run an A/B test?"

At minimum: long enough to reach your required sample size (use a calculator). Also: at minimum 2 full weeks to capture full business cycles (weekend vs. weekday behaviour differs). Don't stop a test early because one variant is "winning" — early leaders often reverse when more data arrives. Don't run tests indefinitely — if you've reached sample size and the result isn't statistically significant, the test is telling you the change doesn't matter enough to detect.

"What if my traffic is too low to run meaningful A/B tests?"

Low-traffic sites have limited A/B testing capacity. Options: test only the highest-impact elements (headline, offer) and accept longer test durations; use qualitative methods (user recordings, heatmaps, user interviews) to build better hypotheses before testing; pool multiple pages with similar templates for a higher-powered test. A/B testing isn't the only way to learn — it's the most rigorous. But user interviews can generate insights even without statistical significance.

The testing roadmap

A systematic testing programme beats random testing:

Month 1: Establish baseline. Set up analytics to measure your conversion funnel properly. Identify the biggest drop-off point. Form your first hypothesis.

Month 2: Run Test 1 on the biggest lever (usually headline or value proposition). Collect data. Document result — win or loss, and what you learned.

Month 3: Run Test 2. Build on what you learned from Test 1 — if a new value proposition direction won, test two variations of that new direction.

The compound effect: If you run one test per month and win 30% of the time with an average 15% improvement per win — over 12 months, that's 3–4 wins compounding. Starting from a 3% conversion rate: after 12 months of systematic testing, 3% × 1.15 × 1.15 × 1.15 ≈ 4.56%. That's an approximately 52% improvement in conversion rate, applied to the same traffic (illustrative calculation — actual compounding depends on whether improvements are independent and non-overlapping, which varies significantly in practice).

Common A/B testing mistakes

Peeking: Checking results daily and stopping when one version "looks like" a winner — before reaching statistical significance. Peeking produces false winners at an alarming rate.

Running too many tests simultaneously: If the same users see multiple tests, interactions between variants contaminate results. Limit active tests to non-overlapping pages or audiences.

Testing the wrong thing: Testing button colour before testing value proposition is working hard for small gains. Always test the highest-impact element first.

Ignoring business context: A December test result may not apply in March. Seasonality, campaigns, and external events can distort test results. Document the context of every test.

Not documenting losses: Failed tests are more informative than wins — they eliminate wrong hypotheses and make the next test better. Document every test, win or loss.

Back to the £15 headline change

The insurance site's growth team didn't spend months on a rebrand. They spent half an hour writing a new headline and four weeks being patient. The result was £1.9M in additional annual revenue from a change that cost £15. What made that possible wasn't creative brilliance — it was a systematic testing programme that had already produced a backlog of hypotheses, a process for running rigorous tests, and the discipline not to peek at results before the sample size was reached. One team relying on designer intuition would have kept the original headline. The testing team proved it wrong. That's the compounding power of treating every assumption as a hypothesis: each test teaches you something your competitors are guessing about.

Key takeaways

• A/B testing is the only way to know — not guess — what works. Expert intuition about creative and copy performance is right barely better than chance. Testing is the only reliable path.
• Statistical significance matters. Running tests with insufficient traffic produces noise, not signal. Use a sample size calculator before every test.
• Test high-impact elements first. Headline, value proposition, and primary CTA changes produce 10–50× more impact than button colour or font size.
• One variable at a time. Multiple changes in a single test make learning impossible — you can't attribute the result to a cause.
• Document everything. Wins, losses, and inconclusive results all contain information. A testing log compounds over time into real institutional knowledge.