Shopify A/B Testing: How to Test Your Store Step by Step

Who it is for

Stores on Grow or above with steady traffic

Time needed

1 hour to set up, 2 to 8 weeks to run

What you need

Your sessions, conversion rate and one hypothesis

An A/B test on a store with 300 orders a month can run for months and still tell you nothing. That is the part most Shopify A/B testing guides skip, and it is why so many store owners try testing once, see "no clear winner" and give up. The test wasn't wrong. It was too small to detect the change.

So this guide starts with the math, then walks through finding a test worth running, setting it up in Shopify's own Experiments tool, running it without spoiling the data and reading the result. It is the process I use in my CRO work. Shopify settings come from its Help Center as I checked it on 7 November 2026.

Do you have enough traffic to test?

An A/B test compares two groups of visitors. The smaller the difference you want to detect, the more visitors you need, and the lower your conversion rate, the more you need again. The standard sample size formula, at 95% confidence and 80% power, gives these numbers for a store converting at 2%:

Divide the total by your weekly sessions to see how long a test would take. If the answer is more than eight weeks, change your plan: test bigger changes, test a step closer to the purchase where rates are higher (such as reached checkout to completed checkout), or skip formal testing and make researched changes, measured against a baseline.

Step 1: Find something worth testing

Good tests come from evidence, not from a list of "50 ideas". Look for the step where you lose the most people, then work out why.

1. Find the leaky step. In Shopify, go to Analytics > Reports and open Conversion rate breakdown. It shows sessions with cart additions, sessions that reached checkout and sessions that completed checkout. Compare each step with the one before it.
2. Split by device. In GA4, open Explore, choose the Funnel exploration template, add your ecommerce events as steps (view_item, add_to_cart, begin_checkout, purchase) and set the breakdown to Device category. Mobile and desktop often leak at different steps.
3. Find out why. Watch session recordings of that step, read support emails and reviews, and run a one-question survey on the page. My Shopify CRO audit checklist lists what to check at each step.
4. Write a hypothesis. Use one sentence: "Because we saw X, we think changing Y will improve Z for these visitors." If you can't fill in X, you are guessing.

Step 2: Choose one primary metric

Decide before launch which number decides the test. If you pick after the results come in, you will find a metric that "won" by chance.

Step 3: Set up the test in Shopify

Shopify's Experiments are a type of rollout. They need the Grow plan or higher and work with online store themes. Vintage themes aren't supported, and Liquid templates can't be changed in a rollout, so you test what you can build in the theme editor or in a second theme.

1. Prepare the change. To test a different theme, add it to your online store first. To test edits to your live theme, you'll make them inside the rollout.
2. Create the rollout. In your Shopify admin, go to Markets > Rollouts and click Create rollout. Give it a clear name, such as "PDP: delivery line under buy button".
3. Choose Experiment. In the rollout type menu, select Experiment.
4. Add the change. In the Changes section, click Add changes. If you are editing your main theme, save the rollout first. Then either click Edit theme under Prepare changes to your current theme and make your edits in the theme editor, or pick a theme under Replace the current theme. Click Apply.
5. Set the audience. In Audience, the Traffic split defaults to 50/50 between control and treatment. Keep it there unless the change is risky. Traffic allocation sets how many eligible visitors join the test at all.
6. Schedule it. Set a start date and an end date. New Experiments default to an end date 90 days after the start. Set yours to the number of full weeks your sample size needs.
7. Resolve conflicts and save. Shopify flags conflicts with other rollouts. You must fix any marked "Resolve rollout conflicts" before saving.

When to use an app instead

Shopify's tool covers theme, discount, catalog and checkout setting tests. You need an app for tests it can't run: shipping rate and free-shipping threshold tests, template and split URL tests, price tests with profit reporting, and tests on the Basic plan. I compare the options, with prices, in my guide to the best A/B testing tools for Shopify.

If you used Google Optimize in the past, it is no longer an option. Google shut down Optimize and Optimize 360 on 30 September 2023, so old guides that recommend it are out of date.

Step 4: Run the test without spoiling it

• Run in full weeks. Shoppers behave differently on weekends and paydays. Start and stop on the same weekday, and run at least two weeks even if the numbers look clear sooner.
• Don't peek and stop. Results swing early. Stopping the moment one version leads is the most common way to "find" a winner that isn't real.
• Don't change the setup. In Shopify, changing the traffic split mid-test only affects new visitors, which mixes two different tests into one result.
• Avoid sales and launches. A big email campaign or a sitewide sale changes who visits. Pause the test or plan around it.
• Remember consent. Shopify counts only visitors who gave tracking consent, and leaves out anyone who saw both versions. Your results come from a share of your traffic, so your sample builds more slowly than your session count suggests.

Step 5: Read the result

For a theme Experiment, Shopify's rollout analytics show conversion rate over time, bounce rate, reached checkout rate and add to cart rate for each group. Checkout Experiments show checkout conversion rate. Results update a few minutes after each visit, and bot traffic isn't counted.

The Help Center doesn't describe how Shopify calculates significance, so I check the result myself before acting on it:

1. Check the sample. Did each version reach the number of sessions you planned? If not, the result is a hint, not an answer.
2. Check significance. Put the visitors and conversions for each group into any standard A/B test significance calculator. Look for 95% confidence or better on your primary metric.
3. Check order value. A version that converts more but sells smaller baskets can earn less. Compare revenue per visitor too.
4. Check devices. A change can win on desktop and lose on mobile. If it does, consider shipping it to one device only.
5. Write it down. Record the hypothesis, dates, numbers and decision. A test log stops you re-running old ideas.

When the treatment wins, apply it permanently. When it loses or ties, that is still useful: you learned the change doesn't matter, and you can stop arguing about it.

What to test first on a Shopify store

My guide to product page design goes deeper on the first two ideas.

Common A/B testing mistakes

1. Testing without enough traffic, then trusting a result that came from noise.
2. Testing tiny changes. A button color rarely moves conversion enough for a small store to measure.
3. Running several tests on the same page at once, so the results contaminate each other.
4. Calling the test early because one version took the lead in the first few days.
5. Ignoring order value and shipping a "winner" that lowers revenue.
6. Not writing results down, then testing the same idea again a year later.

Shopify A/B testing checklist

Next steps

Take your weekly sessions and conversion rate, work out how long a test would take, and decide whether formal testing fits your store today. If it does, pick the leakiest step in your Conversion rate breakdown report and set up your first Experiment this week.

If you'd like help choosing what to test, send me your store. I'll review it for free and suggest the first tests I'd run. My Shopify CRO service, which includes building and analyzing tests, starts from $500.

Frequently asked questions

Sources

1. Creating a rollout, Shopify Help Center
2. Rollout analytics, Shopify Help Center
3. Requirements and considerations for using rollouts, Shopify Help Center
4. Behavior reports, Shopify Help Center
5. Funnel exploration, Google Analytics Help
6. Google Optimize sunset, Google Optimize Help

Related services and work

Written by Ashraf Siddik

I'm a Shopify developer and CRO specialist. I have built and improved Shopify stores since 2018, completed more than 1,000 projects, and hold a 4.96 average rating across 719 client reviews.

About me · Client reviews