Sample Size

Sample Size

Sample size is the number of observations, usually conversions or clicks per variant, that an ad test needs to collect before its result is reliable enough to act on. It is the bridge between a difference you can see on a dashboard and a difference you can trust, because too few observations let random chance masquerade as a real winner. A common working floor is 50 to 100 conversions per variant, but the exact number depends on how large a difference you are trying to detect and how confident you want to be.

Why It Matters

Sample size matters because it is the single most common reason ad tests produce false winners. With a handful of conversions, the variant that happens to be ahead is often ahead by luck, and scaling it pours budget into a result that was never real. Hitting an adequate sample size is what converts a hunch into a decision you can defend.

The relationship is not linear, which trips up most advertisers. Detecting a small difference, say a lift from 2.0 to 2.2 percent CVR, can require thousands of conversions per variant, while detecting a large one, say 2.0 to 4.0 percent, may need only a few dozen. This is why tiny "optimizations" are rarely worth testing on modest budgets: the sample size needed to prove a 10 percent relative lift is often larger than the entire account can generate before the creative fatigues.

How It Works

Sample size is calculated from three inputs before a test launches: the baseline rate, the smallest lift worth detecting, and the desired confidence and power.

• Baseline conversion rate: Start from the variant's current rate, since lower rates need more observations to read a difference.
• Minimum detectable effect: Decide the smallest lift worth acting on, because smaller effects demand much larger samples.
• Confidence and power: Set the confidence level (commonly 95 percent) and statistical power (commonly 80 percent), which together fix the sample needed.
• Per-variant target: The calculator returns the conversions each variant must reach before the result is trustworthy.

The practical takeaway is that you size the test to the effect, not the other way around. If a sample-size calculator says you need 1,200 conversions per variant to detect your target lift and your account produces 30 conversions a day, the honest conclusion is that the test is not feasible and you should either test a bolder change or consolidate budget to raise daily volume.

A Real Example

A skincare brand wants to test whether a new landing page lifts conversion rate from a 3.0 percent baseline. It decides the smallest lift worth acting on is a move to 3.6 percent, a 20 percent relative gain, at 95 percent confidence and 80 percent power.

A sample-size calculation returns roughly 1,000 conversions per variant. At the brand's current pace of 25 conversions per day per variant, reaching that target would take about 40 days, well past the point where creative fatigue would distort the result. Recognizing the test is not feasible at that sensitivity, the brand widens the target to a 40 percent lift, which drops the requirement to about 260 conversions per variant and a roughly 10-day test duration. It runs the realistic test, reaches significance cleanly, and avoids a month-long experiment that would have decayed before it concluded.

Common Mistakes

How Hawky Helps

Hawky's Performance Agent calculates the required sample size from your baseline rate, target lift, and confidence level before a test starts, so it only calls a winner once each variant reaches the threshold that makes the result trustworthy. It tracks conversions per variant in real time against your KPI, refuses to act on under-powered data, and flags when a test is not feasible at the chosen sensitivity rather than wasting spend on a result the traffic can never support.

The Creative Agent feeds tests with challengers that are different enough to produce a large detectable effect, which lowers the sample size needed and lets experiments conclude before creative fatigue sets in. Because every test's sample target and outcome are written to FeatherDB, Hawky learns how much volume your account needs to read a real difference and sizes future tests to your actual traffic instead of a generic rule of thumb.

Frequently Asked Questions

How big does my sample size need to be for an A/B test?

A common working floor is 50 to 100 conversions per variant, but the real number depends on your baseline rate and the smallest lift you want to detect. Detecting a large difference needs far fewer conversions than detecting a small one. Use a sample-size calculator with your baseline, target effect, and confidence level rather than a fixed count.

Should I count clicks or conversions for sample size?

Size your test on the conversion event that defines your KPI, not upstream clicks, because clicks do not prove the variant changed real outcomes. A test with thousands of clicks but a dozen conversions is still under-powered. Clicks can inform secondary metrics, but the conversion count is what determines whether the result is reliable.

Why does detecting a small lift need such a large sample?

Smaller differences are harder to separate from random noise, so you need many more observations to be confident the lift is real. Moving a conversion rate from 2.0 to 2.2 percent can require thousands of conversions per variant, while a jump to 4.0 percent may need only dozens. This is why testing tiny optimizations on modest budgets is rarely worthwhile.

What happens if I act on a sample that is too small?

You risk scaling a false winner, because with few observations the leading variant is often ahead by chance rather than merit. That sends budget toward an ad that does not actually perform better, quietly raising your blended cost. Always reach the calculated per-variant target and confirm statistical significance before acting.

Quick Takeaway

Sample size is the number of conversions per variant a test needs before its result is trustworthy, sized from your baseline rate and the smallest lift worth detecting. Set it before launching and pair it with statistical significance so you never scale a winner that was just luck.

Scaling test winners off a handful of conversions? Ready to hire your first AI performance team? Book Demo