Which AI-Assisted Shopify Orders Can See Your Thank-You Survey?

Monday morning, your AI shopping poll has six replies and Shopify shows 60 orders attributed to AI channels. Did nine out of ten buyers ignore the question? To answer that, you need to know which customers had a chance to see it. A buyer can discover your product in an assistant, finish the purchase on another screen, and leave without ever encountering the placement you configured.

Walk the purchase paths that bring you orders

Start with the AI channels and checkout options actually available to your store. Shopify describes ChatGPT as a referral path to the online store checkout. For some other channels, eligible merchants can offer direct checkout within the channel. Availability and merchant settings affect the route a shopper takes. Shopify's agentic storefronts guide

Create one row in a worksheet for each route you can identify. Useful fields are channel, device, checkout destination, first confirmation screen, survey visible, successful test response and test date. Distinguish a buyer who clicks a recommendation and checks out immediately from a buyer who comes back through a branded search. You may be able to test the first route and only hear about the second in a survey answer.

Run a fresh test order where practical. Record what appears when payment finishes, then separately open the order link as a returning customer. Shopify distinguishes the initial Thank you page from the Order status page shown on subsequent visits. Someone checking the same link twice may accidentally test the second page twice. Shopify's page behavior reference

Count three different things

Orders in scope are the orders you intended to study. Define channel, date, market and order type in advance. Orders with a verified survey exposure are orders for which a reliable measurement recorded that the question appeared. A successful test confirms the placement can work on that route. It doesn't give you an impression count for real orders. Orders with a response are distinct orders for which someone answered the question. Count an order once if you offer more than one opportunity to respond.

For example, imagine a homewares store with 120 weekly orders across two purchase paths. Ninety went through a path where the team verified the thank-you block. The team hasn't verified the other 30. Eighteen distinct orders produced replies. The merchant can report 18 replies per 120 in-scope orders, or 15%. It can also report that the tested path represented 90 of 120 orders. Calling either figure a completion rate among people who saw the question would require unique exposure data.

Investigate gaps before editing the question

An entire checkout path with zero answers merits a placement test. Start with a fresh matching order, check the saved block in the checkout configuration in use, and make sure the order qualifies under your survey's targeting rules. Confirm that the survey accepted the test answer. Use the thank-you page setup walkthrough if the first confirmation screen is missing its block.

When a route is unavailable for testing, label it “unverified” and keep its order count visible. If you collect responses through a separate post-delivery research email, report them as their own cohort. The elapsed time and the decision to open an email change who you hear from. Put the coverage note next to any agentic commerce performance scorecard so the next person reading it knows which orders sit behind the number.

For the Shopify-hosted pages in your test plan, configure a survey with OrderSurvey on the Shopify App Store.

Frequently asked questions