What to Do With Your First 20 Shopify Survey Responses

Use your first survey responses to find a concrete question to investigate, check it against other evidence, and choose one proportionate action. You can learn from a handful of detailed answers without turning them into claims about every shopper.

Twenty responses is a convenient worked example, not a minimum sample size or a guarantee of reliable percentages. A reproducible broken link can be fixed after one report. A decision to move a large advertising budget needs much more evidence than a few remembered discovery sources.

Start with one question and a clear count

Choose the question you want to review, such as “Was there anything that almost stopped you from ordering?” Note the survey, wording, audience, page placements and review dates. Keep a first-order survey separate from one shown to all buyers.

In OrderSurvey, open Responses to read individual answers and Analytics for the dashboard. Both are available on Free. The dashboard also stays visible before the first response, with setup guidance above it. If you have no answers yet, use the setup walkthrough to check the Thank you and Order status placements; zero answers alone does not show whether the problem is setup, eligibility, traffic or participation.

For the example below, “20 answers” means 20 distinct orders that answered one single-choice question after known tests and duplicates were removed. It does not mean 20 exported rows, 20 unique people or 20 total store orders. One order can answer several questions, and a repeat customer can place several orders.

Read optional follow-ups alongside the original answer. Keep missing details and unclear replies visible. If only six orders answered a follow-up, report that detail against six, not all 20. The answer-categorization guide covers a repeatable way to organize open text.

Worked example: a concern you can check today

These figures and comments are invented for illustration. They are not results from an OrderSurvey merchant. Assume the same question, audience and answer choices were used throughout the review period.

Delivery information was selected by 6 of 20 answering orders, or 30%. That describes this group of answers. It does not show that 30% of all shoppers struggled, that six sales were lost, or that delivery information caused checkout abandonment. Everyone in this example completed an order.

Suppose three of those six orders also supplied comments: two could not find an estimated delivery date, while one was unsure when an item would leave the warehouse. These are different questions. Read them before deciding that the shipping price is the problem.

Now inspect the affected product pages on a phone. Check whether the delivery estimate is visible and whether dispatch time is distinguished from arrival time. Compare with relevant support questions, without copying customers' personal details into a shared report.

If the information is missing or contradictory, you have a specific issue to correct. If it is clear, investigate further: perhaps the question refers to a different product, location or earlier version of the page. The survey tells you where to look; the inspection determines what you can substantiate.

Match the action to the strength of the evidence

Early feedback is most useful when the next action is explicit and its cost is proportionate.

For a small batch, showing 6 of 20 is usually more informative than displaying “30%” on its own. If one answer in a fixed batch of 20 moves to another category, the share changes by five percentage points. Small shifts can look dramatic when the counts are hidden.

Separate feedback from a representative estimate

A voluntary post-purchase survey reflects the buyers who saw it and chose to answer. People who skipped it, were ineligible or left without purchasing may have different experiences. Collecting more answers does not, by itself, remove that selection problem.

AAPOR's guidance on opt-in surveys and uncertainty explains why conventional margins of sampling error cannot simply be attached to nonprobability samples. For an early merchant review, describe the observed answers and the limits rather than adding an unsupported confidence claim.

Keep the decision in view. Checking whether a delivery estimate exists does not require estimating the opinions of every shopper. Claiming that a channel drives a particular share of total sales does. The channel-report guide explains the additional order-matching and interpretation work.

Keep a short action record

Write down four things before making a change:

1. Observation: Six of 20 answering orders selected delivery information; three supplied details.
2. Check: The team inspected the relevant mobile pages and found the delivery estimate absent on two of them.
3. Action: The ecommerce owner adds accurate delivery information to those pages and records the date.
4. Next review: Check that the information is visible, then review another defined period of answers alongside related support questions.

This is still an illustrative example. If the number of delivery complaints falls afterward, consider changes in products, traffic, promotions and who answered before crediting the edit with an improvement. A before-and-after comparison alone does not isolate its effect on conversion.

Keep the survey wording stable during the comparison where practical. If you need to change it, record the new wording and date. Pew Research Center's question-design guidance explains how wording, answer choices and question order can affect answers. Use the survey change log to keep the two versions distinguishable.

Start on Free; choose Unlimited when the work needs it

You can read and review an early batch in the app on Free: unlimited surveys and 100 responses across the store per rolling 30 days. Creating another survey does not create another allowance.

Choose Unlimited at $24 USD/month when you need CSV export for a repeatable spreadsheet workflow, more response capacity, configured low-score NPS webhook alerts or priority support. Unlimited does not automatically interpret comments or turn a small voluntary sample into proof about all customers.

Install OrderSurvey to start with one useful question. Dennis can personally help with setup, verify both page placements with your permission, and help you decide what to inspect when the first answers arrive.

Frequently asked questions