Control Groups & Hold Out Groups

The cardinal rule of direct marketing is to include a control group. Without it, you will never know whether customers purchased your product because of this marketing effort, or because of the billboard ad, the radio spot, a friend’s suggestion, an in-store brochure, or because Elvis told them to. There is one exception. If you have an air-tight fulfillment set up, whereby the customer can only purchase the product through your channel, e.g. a special 800#, then you don’t need to hold out a sample; you can be certain that every sale came from your effort (except the referrals from Elvis).

If you sell a product or service through multiple channels, you will be wise to hold out a random sample from your marketing campaign. After the campaign, measure the purchase rate for the hold out group; that’s the benchmark. If 0.80% of the holdouts buy the product during your campaign, then the first 0.80% of your responders to the mailed group don’t count. The “lift” from your marketing efforts is anything above this.

It’s crucial that this control group be representative of the marketed group in every way, except that they won’t receive your offer. If the control group is homeowners and renters, and the marketed group is just homeowners, you’ve got an invalid test that you cannot trust. If the control group is all P$ycle segments, and the marketed group is just the most affluent P$ycle segments, you’ve cheated again, and you can’t trust your test results. You will never know how much of your lift came from your incredible offer, and how much of that same lift came from measuring two different audiences (unless Elvis tells you).

One more caution about control groups (and any market cell, actually): you need to be sure you have enough sample size to trust your results. If you flipped a coin three times, and got two heads, would you be comfortable assuming that the next one hundred flips would give you 66 heads and 34 tails? Nope, you would want lots of flips before you started inferring anything about the weighting of that coin. See Sample Size.

For all your marketing analysis efforts, check out our handy analytics tool!

(Choosing your list) ... (Building a pro forma) ... (Creating your offer strategy)
(Sample size) ... (Hold out groups) ... (Segmenting your customer list)
(Creating selection criteria) ... (Too many exclusions) ... (Evolution of targeting schemes)
(Do you need a response model?) ... (Model gains table) ... (Back end analysis)
(DM analytics tool)