What is A/B Testing? Full Guide + Examples that Work

3. Establish a sample size

Your sample size should be large enough to garner statistically significant results, meaning the outcome is driven by a real difference between versions, not by chance.

With too small a sample size, your results can be misleading and won’t produce accurate insights.

The ideal sample size varies by business, but you can start by reviewing your typical weekly web traffic and audience behavior.

Beene recommends A/B testing tools like Optimizely or Qualtrics to plug in those traffic specs and help you calculate an appropriate sample size for your test automatically.

Tools like Google Analytics and heatmaps can also help you understand traffic patterns before determining your ideal sample size.

4. Identify your timeframe

It’s important to be realistic about timelines for A/B testing. Two weeks is a good starting point for A/B testing, but not every business will fit this schedule.

“[Timelines] will vary for every company,” Been explains. “A smaller start-up may not have the time or resources to put toward more expensive testing.”

She adds that, while SMBs can still A/B test, they will likely have to use smaller budgets and make conclusions more quickly, even if that means sacrificing the quality of data.

How do you know if you’re wasting too much of your budget versus collecting just enough data?

Some marketers recommend halting or waiting to test if you’re working with less than 1,000 monthly conversions. But sometimes we can gather enough data with less if the results demonstrate statistical significance.

For server-side tests or experiments involving algorithms, timelines may vary depending on how fast the system can collect statistically meaningful data.

5. Assess A/B test results

At the end of your A/B testing period, dive into the results to examine how the different variations measured up against your initial metrics and goals.

“Consider the difference in performance between the two variations when assessing your A/B test results,” says Beene.

“If one variation only slightly outperforms the other, you may want to consider some further testing or proceeding with caution when using those learnings.”

And if one variation outperforms the other in a landslide? That’s your clear winner.

“You can confidently determine that applying that strategy on a large scale in your campaigns should lead to improved results,” says Beene.

Always remember that A/B testing is iterative — each test builds on your last insight and helps refine your future experiments.

A/B testing best practices

A/B testing helps you achieve a stellar landing page, ad, or web page that not only resonates with your audience but also drives conversions.

Next, we share some of Beene’s pro insights for boosting your confidence level in A/B testing even more:

1. Only test one element at a time

Changing too many variables at once is one of the most common A/B testing mistakes, and it often leads to inconclusive data.

“While it may seem more efficient on the surface to test a few different variables at once, when you look back at performance, there will be no way to clearly determine what actually moved the needle,” says Beene.

In other words, multiple elements in an A/B test could muddle your data:

“You don’t have clear learnings to take away and use in future strategy decisions,” says Beene.

Testing only one element at a time gives you clear, actionable insights into what works and doesn’t work for your audience. But it’s a delicate balance; you also need enough data to assess that element’s performance.

2. Prioritize data over a specific timeline

Most marketers swear by the two-week timeline as a minimum for A/B testing. While we’ve seen enough data within two weeks to gain actionable insights for a client, that’s not always the case.

That’s because it’s not so much about the length of your A/B test as it is about the quality and volume of data collected.

“For all companies, it’s important to ensure you have enough data (whether that be from a large budget or a smaller budget over a longer period of time) to pull statistically significant findings, with a clear winner,” says Beene.

“Otherwise, there is still some guessing involved when applying your findings on a larger scale.”

3. Act fast to avoid wasted ad spend

“A great way to ensure you’re using your budget efficiently while testing is to know when you have a clear winner from the test and act quickly,” says Beene.

“If you have spent significant budget on a test and one variant is outperforming the other, pausing the underperformer and reallocating budget as soon as you have those learnings is crucial.”

3 A/B testing examples

Before you set goals and test hypotheses, take a look at some A/B tests our team conducted. These examples helped us make informed decisions that yielded impressive results for our clients’ marketing campaigns.

Boosting YOY revenue for 686

Our ecommerce winter apparel client, 686, struggled to define their unique selling points and customer personas.

We analyzed and re-optimized their product feed and Google shopping campaigns, and tested new high-res photos, prices, and optimized product pages to see how they performed.

The data showed us which campaigns attracted the most high-quality leads. We launched and tested remarketing campaigns, adding additional touch points to better reach already interested customers.

Results: 562% increase in year-over-year revenue.

Getting more leads and staying within budget for Peninsula General

Our insurance client, Peninsula General, wasn’t content with their 15% ROAS. To reduce their CPA dramatically and rev up the ROI, we created and tested unique selling points and customer personas that their competition had missed.

We A/B tested new ads to see which clean, user-friendly designs drove the most conversions.

Results: Doubled ROAS while staying within budget.

Bringing in the leads for Zephyr

SaaS platform Zephyr wanted to scale their market share and increase their user base. The problem? Their fast-paced environment with siloed marketing teams created mixed messaging.

We knew immediately that the data would talk. The first thing it told us was that they were draining leads from an ineffective landing page.

By redesigning and A/B testing the landing page, we got the data needed to streamline the lead form and optimize the landing page to operate more efficiently — and improve user experience.

Results: 100% increase in lead volume and 80% reduced CPA.

The takeaway

Whether you’re optimizing page layouts, enhancing a mobile app, or improving a product page, A/B testing helps marketers compare test variations in real time to see what truly resonates with their audience.

But the hardest part about A/B testing is the time and effort it takes to collect and interpret results. On top of that, brands struggle to pick the right variables to test and key performance indicators (KPIs) to track.

This is where the expertise and dedication of a top-3% digital marketing agency bring serious value.

HawkSEM has decades of experience designing and conducting A/B tests for large and small businesses across the finance, ecommerce, travel, insurance, B2B, SaaS niches, and beyond.

In other words, we know what to look for and how to make the most of an A/B testing series.

Ready to unleash the full potential of your A/B tests? Let’s talk.

This article has been updated and was originally published in December 2023.