Did the postcard earn a second mailing? A Shopify holdout and survey

A postcard has a QR code, a discount, and a deadline. After the mailing you have three numbers: scans, code uses, and people who bought. Your printer wants to know whether to reserve another run. The business question is how many more households ordered because you mailed the card. A fourth number, how many remember the card, explains something different.

Set aside a comparison group before the list goes to print

Start with one audience, such as households that previously purchased a full-size candle and have been quiet for a set period. Deduplicate at household level before assignment so two people at the same address cannot land on opposite sides. Randomly assign eligible households to receive the card or to a holdout. Save the assignment, offer, cost, and observation window before mailing.

Keep ordinary email, SMS, pricing, and inventory access comparable across groups. Decide how to match a purchase to an assigned household, how to handle returns, and how long to allow for delivery. Count once if your metric is purchasing households. Keep unmailed addresses in the assigned group for the primary analysis if you discover delivery problems after randomization. Record the mailing issue separately.

Put a neutral question after the purchase

Ask buyers in both groups the same thing: “What reminded you to order from us this time?” Suitable choices include “Running low,” “Email or text,” “Something in the mail,” “A person I know,” “Browsing the store,” and “Something else.” A person who picks mail can answer “What do you remember about it?” without being shown your offer wording first.

Use a dedicated QR code and landing page to see how people who scanned the card behaved. Shopify campaigns support short links and QR codes for this purpose. A household that scanned on a phone and later ordered on a laptop can still appear in the assigned-household comparison. Shopify campaign setup.

Work the economics before the renewal call

Suppose 150 of 5,000 mailed households bought, compared with 120 of 5,000 holdout households. The observed rates are 3.0% and 2.4%, a difference of 0.6 percentage points. At the holdout rate you would have expected 120 purchasing households in the mailed group. The observed difference amounts to 30 households.

For this example, assume one eligible order per purchasing household, $30 contribution per order after variable costs, and $0.70 per mailed piece including postage. The point estimate gives $900 in additional contribution against $3,500 in mail costs. Break-even requires 117 additional orders, before fixed overhead. Show the counts alongside the estimate. Random assignment improves the comparison, but chance still affects the observed difference.

Now suppose 40 responding buyers recalled the postcard. That detail can inform the next creative brief even though adding 40 to the 30 would double-count evidence about the same campaign. If people remembered the refill reminder but struggled to use the code, try a simpler offer in a smaller follow-up test. If the repeat-order economics depend on a second purchase, define that follow-up window and measure it in both groups before using lifetime value to justify a larger mailing.

Run the thank-you page question with OrderSurvey on the Shopify App Store. For other offline spend, the podcast measurement worksheet covers code overlap, while the survey revenue guide shows how to carry order value into the analysis.

Frequently asked questions