Shopify glossaryGlossary4 min read
What Is A/B Testing? How to Run Split Tests on a Shopify Store
A/B testing shows two versions of a page to different visitors and measures which one performs better. Here is how it works, the math behind a result, and how to run one on Shopify.
Key takeaways
- A/B testing splits visitors at random between a control (A) and a changed version (B), then compares a goal such as conversion rate or revenue per visitor.
- A difference only counts if it's statistically significant. A 15% lift on a few hundred orders can still be chance.
- Shopify's built-in Rollouts tool runs experiments on themes, checkout and accounts, catalogs and discounts, on the Grow plan or higher.
- Low-traffic stores often can't detect small lifts. They get more from research and bigger changes than from many small tests.
- Who it is for
- Stores ready to test changes
- Time needed
- 4 minutes to read
- What you need
- Traffic and a clear hypothesis
A/B testing is a controlled experiment in which you show two versions of a page or feature to randomly split groups of visitors, then measure which version better achieves a goal, such as purchases. Version A is the control, your current page. Version B has one planned change.
It's the part of CRO people hear about most, and in my work it comes last, after the research that decides what's worth testing. A test without a strong hypothesis just measures noise. Shopify's Rollouts details below were checked on the Shopify Help Center on 7 November 2026.
How an A/B test works
- Hypothesis. "Shoppers on mobile miss the delivery date, so showing it under the price will raise add-to-cart rate." It names a problem, a change and a metric.
- Split. Visitors are assigned at random, usually 50/50, and keep seeing the same version on return visits.
- Primary metric. Pick one before you start. I use revenue per visitor or conversion rate, with add-to-cart rate as a supporting signal.
- Run time. Run whole weeks, at least two, so weekday and weekend shoppers are both counted.
- Read the result only when the planned sample is reached.
A worked example, with the math
A product page test over four weeks. The figures are illustrative.
| Version | Sessions | Orders | Conversion |
|---|---|---|---|
| A: control | 15,000 | 300 | 2.0% |
| B: variant | 15,000 | 345 | 2.3% |
Relative lift = (B − A) ÷ A = (2.3% − 2.0%) ÷ 2.0% = 15%. That looks like a clear win. But a standard two-proportion test on these numbers gives a p-value of about 0.07, above the usual 0.05 cut-off. In plain English: there's still a real chance this gap is luck. The right call is to keep the test running to its planned sample, not to ship B.
How much traffic a test needs (illustrative)
- To detect a 10% lift~78,000Sessions per version, from a 2% baseline.
- To detect a 30% lift~8,700Sessions per version, same baseline.
This is why small stores struggle with A/B testing. At 20,000 sessions a month, finding a 10% lift takes most of a year. Testing bigger, bolder changes is the practical answer.
How to A/B test on Shopify
Shopify now has a native option called Rollouts:
- Where:
Markets > Rolloutsin the Shopify admin. Choose the Experiment type. - Plan: your store must be on the Grow plan or higher to run experiments.
- What you can test: the online store theme, the checkout and accounts configuration, catalogs and discounts.
- Defaults: a 50% control and 50% treatment split, and an end date 90 days after the start. You can change both.
- Metrics: theme experiments report conversion rate over time, bounce rate, reached checkout rate and add to cart rate. Checkout and accounts experiments report checkout conversion rate.
Third-party testing apps add features such as price tests, split URL tests and revenue-per-visitor goals. I compare them in Shopify A/B testing tools, and the full setup is in my guide to A/B testing on Shopify.
Common mistakes
- Stopping when it looks good. Checking daily and stopping at the first green number produces false winners.
- No hypothesis. "Let's try a green button" teaches you nothing even if it wins.
- Judging on conversion alone. A variant can raise conversion and lower order value. Revenue per visitor catches that.
- Testing during a sale. Black Friday shoppers behave differently, and the result may not hold in March.
- Testing pages with little traffic. Start where the traffic and money are: product pages, cart and collection pages.
Related terms
- Conversion rate optimization: the wider practice that A/B testing belongs to.
- Heatmap: a research tool that helps you find what to test.
- Multivariate testing: tests several changes in combination, and needs far more traffic.
- Statistical significance: how unlikely the result would be if the change had no real effect.
What to do next
- Check your traffic. Note monthly sessions and orders on your top pages to see what size of lift you can detect.
- Write three hypotheses from customer questions, reviews and heatmaps.
- Run your first Rollouts experiment on the change with the biggest expected effect, or have me plan the program. My Shopify CRO service starts from $500.
Free tools
Check your own store
Free, no sign-up. Each one runs a part of this guide for you.
Frequently asked questions
What is A/B testing in simple terms?
A/B testing shows two versions of a page to randomly split groups of visitors and measures which one gets more of a goal, such as purchases. Version A is the current page and version B has one planned change.
Can you A/B test on Shopify without an app?
Yes. Shopify's Rollouts tool, under Markets, can run experiments on themes, checkout and accounts, catalogs and discounts. Your store must be on the Grow plan or higher.
How long should an A/B test run?
Until it reaches the sample size you planned, and for whole weeks, at least two. Stopping early because a result looks good is the most common way to get a false winner.
How much traffic do I need for A/B testing?
It depends on your conversion rate and the lift you want to detect. From a 2% baseline, spotting a 10% lift takes roughly 78,000 sessions per version, while a 30% lift needs about 8,700.
What is an A/B testing example on a Shopify store?
Showing the estimated delivery date under the price on mobile product pages to half of visitors, then comparing add-to-cart rate and revenue per visitor between the two groups.
Sources
- Rollout types, Shopify Help Center
- Rollouts analytics, Shopify Help Center
- Shopify analytics fields, Shopify Help Center
Related services and work
I'm a Shopify developer and CRO specialist. I have built and improved Shopify stores since 2018, completed more than 1,000 projects, and hold a 4.96 average rating across 719 client reviews.
More guides
-
Why Shopify Stores Lose Sales on Mobile: A Practical CRO Checklist
Mobile visitors decide fast, with one thumb and plenty of distractions. Here is how to find the leaks on a Shopify store and fix them in the right order.
-
Shopify CRO Audit Checklist for DTC Brands
The checklist I use to audit DTC Shopify stores, from checking that the data can be trusted to deciding which fixes to ship first.
-
When to Build a Custom Shopify App Instead of Adding Another Subscription
Another subscription is the quickest fix, until you are paying for five apps that still don't match how your business works. Here is how to decide.
-
How to Find Out What Theme a Shopify Store Uses
Every Shopify store names its theme in the page code. Here is where to look, how to read what you find, and what it means for your own theme choice.
-
How to See What Apps a Shopify Store Uses
Apps leave traces in a store's page code: scripts from their own servers, app embed folders and pixel settings. Here is how to read them, and what to do with the list.
-
How to Name a Shopify Store (and Check the Name Is Free)
A store name has to be easy to say, free as a domain and a handle, and clear of other brands. Here is the process I use with clients, and what Shopify lets you change later.
