How to A/B test a Shopify thank-you page survey question

A Shopify merchant wants better discovery data from the thank-you page. Their first idea is to replace “Where did you first hear about us?” with “What brought you here today?” Either question could earn more taps, but each measures a different point in the purchase journey. If your goal is to test wording, hold the underlying question steady. Otherwise the winning version could answer the wrong business question beautifully.

Write the experiment on a single card

Consider a store comparing two introductions above the same question. Version A says “A question about your order.” Version B says “Help us learn where customers discover us.” Both lead into “Where did you first hear about our store?” The options, their order, the follow-up, the placement and the audience stay identical.

Write a prediction before launch. In this example, explaining the purpose might help more shoppers give an answer the marketing team can classify. Define a classifiable answer up front: a named source, including a write-in that can be mapped under a documented rule. Keep “I don't remember” available and count it in total completions. Report that choice separately so an increase in guessed answers does not masquerade as better source coverage.

Record the primary outcome as assigned eligible orders with a classifiable first answer divided by all assigned eligible orders. Make the unit explicit. If a person can order repeatedly during the test, either limit this example to first orders or assign consistently by shopper and account for repeated observations in the analysis. See the denominator worksheet for matching counts.

Assign both versions during the same period

Use a setup that can randomly assign eligible shoppers and retain the assigned version through the response. Verify one test order per version and check that assignment records include orders that never submit. A survey response export alone misses those silent shoppers. Plan the sample size using your baseline rate and the smallest difference worth acting on. Record the stopping rule before launch. The UK's Government Digital Service explains the underlying method in its A/B testing guidance.

A small store without random assignment can still learn. Run the two versions in comparable periods, log sales and campaign changes, and label the result a before-and-after comparison. A creator mention on Tuesday may change both the volume and the clarity of answers. It deserves a line in the analysis. A few buyer interviews may resolve a confusing phrase faster than waiting for the store to accumulate enough orders.

Interpret a result with a pencil

Imagine 1,000 eligible first orders were assigned to each version. Version A produced 120 classifiable answers and B produced 140. That is 12% against 14%, a gain of two percentage points, or 20 additional classifiable answers per 1,000 assigned orders. Under a simple independent-orders comparison, an approximate 95% interval for that difference runs from about minus one to plus five percentage points. A small decline and a worthwhile increase are both consistent with these illustrative counts.

Before deciding, examine the actual answers. Did B produce more named sources across categories, or an unusual jump in whichever option appears first? A shift in the mix can uncover a wording problem worth investigating even when the difference in the main outcome is uncertain. Retain the variant, dates, counts and decision so a teammate can interpret next quarter's trend.

OrderSurvey on the Shopify App Store provides the thank-you page survey block. Random assignment and assignment logs are separate requirements for this experiment. For the shopper-facing wording beneath either introduction, use the discovery question options guide.

Frequently asked questions