What Was on the Shortlist? A Shopify Poll for AI Product Comparisons

A grinder merchant thinks its closest competitor sells a grinder that costs $20 less. A buyer comparing products with an AI assistant may be weighing something entirely different: keep the hand grinder already on the counter or buy an electric one quiet enough to use before the household wakes up. A comparison page about price will do little for that shopper.

The shortlist is worth asking about because the alternatives reveal what the buyer actually had to decide. Shopify Catalog can supply assistants with product titles, descriptions, options and other attributes. Your survey can help you decide which of those facts deserves a clearer presentation. Shopify Catalog and product discovery

Ask about the next-best choice

If you already ask whether a buyer used an assistant, direct the follow-up to people who say they used it to compare products. If your store has a small number of AI-assisted buyers, ask the comparison question of all buyers for that product and describe it as a broader product study. Our AI shopping survey question guide gives you the first version.

Ask “If you hadn't chosen this grinder, what would you most likely have done?” Adjust the noun to fit the purchase. Offer choices that lead to different merchandising decisions:

• Bought a different grinder from this store
• Bought a grinder from another brand
• Kept using the grinder I already have
• Bought pre-ground coffee instead
• Put off the decision
• Something else or unsure

Ask one optional follow-up: “What detail mattered most in making that choice?” Leave it open the first time. A prewritten list that says grind consistency, speed and price makes it easy to overlook the customer grinding coffee beside a sleeping child.

Turn a shortlist into a page customers can use

Imagine 24 electric grinder buyers answer. Ten would have kept their current grinder, seven would have bought another model from the merchant, four would have bought from another brand and three would wait. Of the ten who would have kept their grinder, six describe early mornings or shared walls. Of the seven who considered a sibling model, four mention cleaning.

Those two groups deserve different answers. On the product detail page, explain how the motor sounds under real conditions. An audio clip should identify the bean load, microphone distance and room so customers can judge what they're hearing. For the sibling models, show how the burrs come out, which parts can be washed and how long reassembly took in your demonstration. Keep measurements consistent across the comparison so a shopper can use them side by side.

Check whether the comparison answered the hard part

After the content update, keep the same buyer population and run “Which detail, if any, did you still have to look up somewhere else?” Leave room for “I found everything I needed.” Watch what people actually name. A new cluster of questions about replacement parts gives the merchant a better next edit than another round of copy about how quiet the motor is.

Avoid presenting these counts as competitor win rates. Every respondent chose your grinder, and your sample can say very little about people who kept their old one. Include those buyers through interviews or a separate visitor study if your decision is whether to develop a new model. When the people who leave are the group you need to hear from, the Shopify exit survey guide covers that different sampling problem.

Run the shortlist poll on the thank-you page for the product you are studying with OrderSurvey on the Shopify App Store.

Frequently asked questions