Your team has two ideas for the first page of a collection: keep proven sellers near the top, or give available new arrivals more room. A collection A/B test can help you choose, but only if shoppers actually receive different orders and the report measures the decision you care about. Agree on those checks before trusting a winner badge.
Write the decision before opening the dashboard
Start with a sentence your merchandiser can challenge: “Giving available new fragrances more exposure will increase purchases among visitors to this collection.” That is a hypothetical test idea, not a promise about new products.
Keep the current strategy as A and describe exactly what changes in B. If B also gets new photography, a discount and a different product-card layout, you are evaluating that whole package. You cannot credit the sorting change alone.
Shopify's collection ordering documentation distinguishes built-in options including Best selling, which uses the all-time number of orders containing a product, and manual ordering. Record the actual starting option. “Our usual order” is too vague to reproduce later.
In Swift AI Collections, I built a comparison between two sorting strategies. With the theme extension installed, visitors are assigned on the storefront and both orders are served at the same time. Without it, the project uses alternating time periods grouped by day of week. Its Bayesian analysis reports the probability that B beats A, an interval for the lift and the expected loss of choosing the wrong version.
Those are documented implementation choices. The brief and checks below are my proposed way to review a test for your store; they do not describe a measured sales improvement from that project.
Fill in a test brief your team can use
Choose a primary measure that answers the buying question. For the fragrance example, that could be the share of eligible collection visitors who buy within an agreed observation window. Include visitors who do not click a product: the ordering itself may affect whether they click.
| Decision | What to record before launch |
|---|---|
| Change | A keeps the current strategy; B changes only the agreed treatment of new arrivals |
| Eligible visitors | Which collection, markets and devices are included, and when a visitor enters the test |
| Success measure | Numerator, denominator and how long purchases can count after the visit |
| Business limits | Stock eligibility, campaign pins and any margin condition both versions must respect |
| Delivery | How assignment stays consistent, and what happens when a shopper selects another sort |
| Decision rule | The smallest improvement worth adopting, acceptable downside and planned review points |
| Exit | Who can stop the test and how the previous experience is restored |
Ask the analyst to estimate feasibility using that collection's eligible traffic and purchase history. Agree on a maximum run window and an inconclusive outcome before starting. A large catalog does not by itself tell you how much usable evidence this particular collection receives.
Use the collection sorting brief first if your team still disagrees about which products should be eligible for the first page. An experiment needs defined alternatives.
Check delivery before comparing sales
Ask for a demonstration of both groups on the storefront. Record the assigned group, the order actually displayed and the event that says the shopper saw it. An assignment recorded before the alternative order successfully loads is not enough evidence of delivery.
Try a reload, pagination, filters and the customer's sort selector. Decide in advance how a shopper choosing price order is handled. Don't remove those shoppers later just because their behavior makes the result less attractive.
Follow an agreed test purchase through the reporting. Check that repeat events do not count the purchase twice and that its group matches the recorded collection visit. Test missing consent separately: Shopify's pixel privacy documentation says an app pixel loads only when the visitor has permitted all settings it declares as required. Ask which visitors and purchases the report can observe; an unobserved purchase is not evidence that someone did not buy.
If the implementation alternates collection orders over time, ask to see the schedule. Comparing Monday's A with Saturday's B leaves the day and the order entangled. Swift AI Collections groups its alternating periods by day of week, but that does not make them simultaneous visitor groups. Record promotions, stockouts and other changes across the periods, and have the analysis account for the time-based design.
Read the Bayesian result in three parts
“Bayesian” describes an analysis that combines starting assumptions with observed evidence. You don't need its equations to ask useful questions. The Statsig Bayesian documentation explains the terminology; it is a reference for these concepts, not a claim that Swift AI Collections uses Statsig.
- Chance to beat A: under the model, how likely is B to improve the chosen measure? It does not tell you whether the improvement is worth implementing.
- Credible interval: the range containing a stated share of the model's probability for the effect. A range spanning both loss and gain leaves uncertainty about direction.
- Expected loss: the model's average downside from choosing a version when another would be better. Check the units; it is not a worst-case loss limit.
Ask what starting assumptions, or priors, were used and whether reasonable alternatives change the decision. The probability is conditional on the model and evidence, not a guarantee about next month's sales.
Agree what to do when the answer is unclear
Imagine a hypothetical report where B has a positive estimated effect, but the uncertainty range still includes a loss your team considers unacceptable. Don't translate that into “B works.” Record that the available evidence does not yet support the change under your agreed rule.
At each planned review, separate three decisions:
- Adopt: the evidence meets the written benefit and risk criteria, delivery checks pass, and the business limits hold.
- Continue: the test remains valid and more observation is feasible within the agreed window.
- Stop: delivery is broken, a business limit fails, or the window ends without enough evidence. Record whether the result was harmful, invalid or inconclusive.
Check operational failures whenever they arise; don't wait for a statistical review to fix a broken cart. But don't change the success measure or keep extending a test solely to find a favorable result. Preserve the original brief alongside the final decision.
The search revenue attribution guide addresses a related distinction: assigning credit to an interaction is different from showing that a change improved outcomes. A collection experiment needs its own comparison, even when an attribution dashboard looks encouraging.
Start with one reviewable test
Pick a collection where your team can explain both orders. Complete the brief, request the storefront demonstration and ask for an example of how the reporting would label an inconclusive result. You can use those checks when evaluating an app or commissioning a custom implementation.
If you'd like help with the test design or storefront delivery, share the collection and proposed change through Shopify search and merchandising enquiries, or book a call. That gives us a concrete decision to work through.
