Holdout Group
Last updated
A randomly selected slice of your audience kept from seeing a campaign, used as a clean baseline to measure the incremental lift your ads truly caused. It reveals the real impact that platform-reported ROAS routinely overstates.
Holdout Group
A holdout group is a randomly selected portion of your audience that is deliberately kept from seeing a campaign, so its behavior becomes a clean baseline for measuring the campaign's true effect. By comparing the conversions of the exposed group against this untouched control, you isolate the incremental lift your ads actually caused rather than the conversions that would have happened anyway. It is the advertising equivalent of a control group in a scientific trial, and it answers the hardest question in media measurement: did the ads cause the sales, or just take credit for them.
Why It Matters
A holdout group matters because standard attribution systematically overstates what ads achieve. Platform reporting and last-click models credit conversions to ads even when the customer was already going to buy, which inflates ROAS and hides waste. A holdout cuts through that by showing what happens with no exposure at all, so the gap between the two groups is the real, incremental result.
The difference is often dramatic. Studies of incrementality testing routinely find that platform-reported conversions overstate true lift by 20 to 50 percent or more, especially in retargeting and branded search where the audience was already primed to convert. Without a holdout, a retargeting campaign can look like a hero while contributing almost nothing on top of organic demand, and you would never know.
How It Works
A holdout group works by randomizing your audience into an exposed group and a held-out control before the campaign runs, then comparing conversion rates between them after it ends.
- Randomize the split: Assign a slice of the audience, often 5 to 20 percent, to the holdout at random so the two groups are statistically identical.
- Suppress exposure: The holdout is blocked from seeing the campaign, while everyone else is eligible for ads as normal.
- Measure both groups: Track conversions in the exposed group and the holdout over the same window.
- Calculate incrementality: The difference in conversion rate between the groups, scaled up, is the campaign's true causal lift.
The integrity of the test rests on the randomization and the suppression actually holding. If the holdout leaks, meaning held-out users somehow see the ads, the baseline is contaminated and the lift reading becomes meaningless. This is why platform-level holdout tools, such as Meta's conversion lift studies, are more reliable than manually excluding an audience that ads can still reach through other campaigns.
A Real Example
A subscription brand spends $40,000 a month on a Meta retargeting campaign that platform reporting credits with 1,000 conversions, an apparent 5.0 ROAS. Suspecting the number is inflated, the team runs a conversion lift study with a 15 percent holdout.
Over four weeks, the exposed group converts at 4.2 percent while the held-out control converts at 3.4 percent on its own, driven by organic demand and brand familiarity. The incremental lift is 0.8 percentage points, which means only about 190 of the 1,000 reported conversions were truly caused by the ads. The real incremental ROAS is closer to 1.0, not 5.0. Armed with the holdout result, the brand cuts the retargeting budget by 60 percent and redirects it to prospecting, where a separate holdout proves genuine incremental lift.
Common Mistakes
| The Mistake | ❌ The Wrong Way | ✅ The Hawky Way |
|---|---|---|
| Leaky holdouts | Excluding an audience in one campaign while other campaigns still reach them. | Using platform lift tools that suppress exposure across the whole account. |
| Holdout too small | Holding out 1 percent, leaving too little data to read lift with confidence. | Sizing the holdout so the control reaches an adequate sample size. |
| Trusting platform ROAS | Scaling a campaign on reported ROAS that never accounts for organic demand. | Measuring incremental lift against a true holdout before scaling. |
How Hawky Helps
Hawky's Performance Agent runs holdout-based lift tests against your real KPI, sizes the control so it reaches an adequate sample size, and logs the incremental result so budget decisions rest on causal lift rather than inflated platform ROAS. It watches which campaigns actually move conversions above baseline and reallocates spend away from the ones that only take credit for organic demand.
The Creative Agent keeps the exposed group supplied with fresh, on-brand creative so the test measures the campaign at its best rather than against fatigued assets. Because every holdout result is written to FeatherDB, Hawky remembers which campaigns proved incremental and which did not, so it stops re-funding channels that a prior holdout already exposed as low-lift.
Frequently Asked Questions
What is a holdout group in advertising?
A holdout group is a randomly chosen portion of your audience that is intentionally kept from seeing a campaign, acting as a control. By comparing conversions in the held-out group against the exposed group, you measure the incremental lift the ads truly caused. It separates real campaign impact from conversions that would have happened anyway.
How big should a holdout group be?
A holdout is typically 5 to 20 percent of the audience, large enough that the control collects an adequate sample of conversions to read lift with confidence. Too small a holdout produces a noisy baseline, while too large a holdout sacrifices reach and revenue. Size it against the conversion volume you expect, not a fixed percentage.
What is the difference between a holdout group and A/B testing?
A/B testing compares two active variants to see which performs better, while a holdout group compares an exposed audience against one that sees no ads at all. A/B testing answers which ad wins, and a holdout answers whether the ads caused incremental conversions in the first place. They solve different problems and are often used together.
Why is platform-reported ROAS often wrong?
Platform reporting credits conversions to ads even when the customer was already going to buy, which inflates ROAS, especially in retargeting and branded search. A holdout group reveals the share of those conversions that would have happened with no ads at all. The true incremental ROAS is frequently far lower than the platform number suggests.
Quick Takeaway
A holdout group is an audience kept from seeing your ads so it becomes a clean baseline, revealing the incremental lift your campaign actually caused instead of conversions it merely took credit for. Size it for an adequate sample size and trust the lift over platform-reported ROAS.
Scaling campaigns on platform ROAS that never proves real incrementality? Ready to hire your first AI performance team? Book Demo
