Your search dashboard shows revenue, but the order contains products the customer never found through search. Before using that number to judge a search project, check which items earned credit, what amount was counted and which purchases the tracking could observe. A mixed basket is a useful first test: search should earn only the credit your reporting rules actually support.
1. Decide what the number should answer
“Did people who used search buy?” and “Which purchased products had a recorded search interaction?” are different questions. The first can describe an entire shopping visit. The second needs evidence connecting an interaction to a particular order line. Neither tells you what the customer would have bought without search.
Start by asking which parts of the storefront the report covers. Shopify's Search & Discovery analytics documentation says its search reports cover the search results page and exclude predictive-search interactions. A report for a custom suggestions panel may therefore describe a different set of activity. Comparing their totals without checking coverage won't tell you which is wrong.
In SwiftSearch, I chose line-item attribution so a searched-for product doesn't claim credit for a browsed product in the same order. I used a seven-day last-touch window, consent-gated tracking and separate totals for each presentment currency—the currency used for that purchase. The dashboard describes the orders its pixel observed, rather than all store sales.
Those are documented project decisions. The worksheet and acceptance tests below are my proposed way to review a store's reporting; they aren't claimed test results from a client project.
2. Follow one item from search to checkout
Ask for a demonstration with a product you can recognize. Search for it, interact with its result and buy it in an agreed test checkout. Have the developer show which recorded action makes the purchased item eligible for credit.
Write down these rules before comparing providers:
- Qualifying action: does a result view, click or add-to-cart earn credit?
- Product match: must the purchased variant match, or does any size of that product qualify?
- Time limit: how long after the action can a purchase qualify?
- Multiple searches: which eligible interaction gets credit when several precede the purchase?
If your agreed rule requires an exact variant match, clicking a 50 ml fragrance and buying its 100 ml version would not qualify automatically. A product-level rule might include it. Either rule needs a clear label and a demonstration; an unexplained match is difficult to audit.
SwiftSearch's seven-day window is a design choice, not a Shopify requirement or a universal best setting. Ask your provider what resets its window and how it prevents the same purchased line from being credited repeatedly to different queries.
If products aren't reaching the cart reliably, start with the search troubleshooting worksheet. Revenue reporting won't explain a missing product or a broken variant picker by itself.
3. Check a mixed basket by hand
Here is a hypothetical order in USD, excluding tax and shipping. Assume the fragrance has a qualifying search interaction and the body wash was found only by browsing. For this trial, the agreed measure is purchased item value after allocated discounts, before later refunds.
| Purchased item | Quantity and price | Discount allocated to the line | Item value after discount | Search credit |
|---|---|---|---|---|
| Fragrance found through search | 2 × $60 | $20 | $100 | $100 |
| Body wash found by browsing | 1 × $20 | $0 | $20 | $0 |
| Total | $140 before discounts | $20 | $120 | $100 |
The expected credit is $100 under these rules. Crediting $120 would include the browsed item; crediting $140 would also ignore the discount.
Ask how order-level discounts are assigned to items, whether tax and shipping enter the measure, and how quantities are handled. If the dashboard reports an amount before discounts, it should say so. Different definitions can be useful, but they should not share an ambiguous “revenue” label.
Keep currencies separate unless the report names its conversion method. Also ask what happens after a refund. A checkout-time amount can remain useful as a historical measure, but it should not be presented as retained sales if later adjustments aren't included.
4. Test what happens when evidence is missing
A checkout event is an observation, not a complete store ledger. Shopify's checkout_completed reference explains that the event can occur on the first post-purchase upsell page rather than the Thank you page. If the page where it should fire doesn't load, the event isn't triggered.
Consent also affects what a pixel can observe. Shopify documents pixel privacy settings and consent updates; ask your developer to demonstrate the relevant behavior in your configured store. A missing observation must not be relabeled as a purchase with no search involvement.
Use this worksheet with your developer or reporting provider. Record the test order reference, currency, expected amount, observed amount and explanation beside each result. Keep test activity identifiable and excluded from ordinary performance reporting.
| Test | What the report should demonstrate |
|---|---|
| Search for one item, browse to another, then buy both | Only the eligible line receives credit under the agreed line-item model |
| Buy two units with a discount | Quantity and the agreed discount treatment reproduce the hand calculation |
| Repeat delivery of the same purchase observation in a developer test | The same order line isn't counted again |
| Purchase after the agreed time limit | The expired interaction receives no credit |
| Decline the consent required for analytics | Tracking respects the setting; missing activity isn't invented |
| Buy in a second currency | A separate total or an explicitly documented conversion appears |
| Refund the eligible item | The report adjusts it, or clearly labels that later refunds aren't included |
Ask for an explanation of unsupported cases before using the dashboard for a spending decision.
5. Use the result without overstating it
Keep three outcomes distinct: an observed purchase with eligible search credit, an observed purchase without it, and a purchase the tracking didn't observe. You may not be able to identify every missing purchase individually; the report should still disclose its coverage limits.
A rise in attributed revenue can justify looking more closely at a query or product. It doesn't establish that search created extra sales. Record promotions, stock changes and changes to the attribution rules alongside comparisons. If the decision depends on proving additional sales, ask for a separate experiment plan instead of treating the attribution total as the answer.
When evaluating native search, an app or custom work, include this reporting trial alongside the shopper tasks. A provider should be able to explain a modest, traceable number just as clearly as a large one.
6. Start with one explainable order
Write your credit rule in one sentence, then work through the mixed basket with your provider. Save the expected calculation and the dashboard result. You can use that record to accept the reporting, request a correction or identify a limitation without replacing your search system.
If you'd like help scoping search reporting or reviewing the connection between search and checkout, send that example through my Shopify search and merchandising enquiry page, or book a call. Bring your current search provider and the number you want to be able to trust.