Incrementality: What It Is and How to Measure the Real Effect of Your Ads

Quick Answer

Incrementality is the share of your results that your advertising actually caused, as opposed to results that would have happened anyway. If 100 people bought after seeing your ads, but 70 of them would have bought without the ads, the ads' incremental effect is 30 sales. You measure it with an experiment: show ads to one group, hold them back from a comparable control group, and compare outcomes. The difference is your lift. Divide the revenue from that lift by your spend and you get incremental ROAS (iROAS). Attribution reports can't give you this, because they only count conversions that followed an ad, not conversions the ad caused. The main methods are user-level holdouts, platform conversion-lift studies, geo experiments and time-based on/off tests. Each needs enough volume to produce a reliable answer.

This guide covers incrementality for advertising in general. For the specific question of whether to bid on your own brand name, with a worked pause test, see our guide to bidding on branded keywords.

What Incrementality Means (In Plain Terms)

Every conversion after an ad falls into one of two buckets:

Advertising always pays for both buckets. Incrementality measurement tells you how big the first bucket is.

The key idea is the counterfactual: what would have happened if the ad had not run? You can never observe both worlds for the same person, so you compare two similar groups instead.

Incrementality vs. Attribution

Both are useful. Think of attribution as the daily dashboard and incrementality as the audit that tells you whether to trust it. Many measurement guides recommend combining them rather than choosing one.

Why Platform Numbers Can Overstate the Effect

Several things inflate reported results:

• Targeting people who were already going to convert, such as brand searchers, past visitors and existing customers
• Credit overlap: two platforms can each claim the same conversion
• View-through and long attribution windows that credit ads for sales that happened to follow an impression
• Platforms report on their own ads, so they have a built-in incentive to report generously
• Invalid traffic that is counted as a click or visit but is not a real customer (covered below)

Incrementality testing is how you check these numbers against reality. Unsupported anecdotes should not substitute for a controlled test on your own campaign.

The Core Formulas

Let p(exposed) be the conversion rate among users randomly assigned to the advertising treatment, including those who did not receive an impression, and p(holdout) the rate among users assigned to the holdout. Comparing only people who actually saw ads can introduce selection bias.

Reported ROAS divides all attributed revenue by spend. iROAS divides only the extra revenue the ads caused. iROAS can be lower or higher than reported ROAS because attribution can also miss effects. It estimates average incremental return for the tested spend; it does not directly establish the marginal return from another dollar.

Methods for Measuring Incrementality

Size geo experiments using baseline variation, the minimum detectable effect, geographic clusters and conversion volume. There is no universal minimum spend that guarantees a reliable result.

How to Run an Incrementality Test: An Eight-Step Runbook

1. Decide what you're asking. Example: "If I turn off this campaign in some regions, how many fewer sales will I get?"
2. Choose one outcome that matters, such as purchases or qualified leads, not clicks.
3. Pick the design from the table above based on your channel, volume and tools.
4. Calculate how big the test must be (see the next section) before spending anything.
5. Build comparable groups. For geo tests, match regions on size and baseline conversion rates. For user tests, randomize.
6. Freeze everything else. Don't change budgets, creative, promotions or tracking in the test or control groups during the test.
7. Run long enough to cover your normal buying cycle and conversion lag, so late conversions are counted. Avoid sales events and launches.
8. Analyze with uncertainty. Report lift, a confidence interval, iROAS and what decision the result supports.

How Big Does the Test Need to Be?

This is where most small tests fail. If the true effect is small and your conversions are few, the result will be too noisy to trust.

Measurement guides describe the standard approach: estimate the baseline conversion rate, the smallest lift worth detecting, the significance level (commonly 5%) and the statistical power (commonly 80%), then calculate the sample size per group.

The table below is our own calculation using the standard two-proportion formula, with a two-sided 5% significance level and 80% power. It shows the number of people needed in each group (exposed and holdout).

What this tells you:

• Smaller lifts and lower baseline conversion rates need much larger groups.
• Detecting a 5% lift on a 1% baseline needs more than 600,000 people per group, which is out of reach for many advertisers.
• These are individual-randomization approximations. Geo clustering, unequal allocation and spillover require different power calculations.

If your test can't be large enough, you can still learn something, but you should expect a wide range of possible answers and avoid overreacting to the point estimate.

Reading Results: A Worked Example (Illustrative Numbers)

Suppose you randomly hold out 10% of an audience.

• Absolute lift = 2.30% − 2.00% = 0.30 percentage points
• Relative lift = 0.30 ÷ 2.00 = 15%
• Incremental conversions = 0.30% × 90,000 = 270
• If each conversion is worth $60 and you spent $12,000: incremental revenue = 270 × $60 = $16,200, so iROAS = 1.35

Now the part many summaries skip: the uncertainty. Using a standard 95% confidence interval for the difference in conversion rates:

• Absolute lift: roughly 0.01 to 0.59 percentage points
• Relative lift: roughly 0.4% to 30%
• iROAS: roughly 0.04 to 2.7

The unadjusted 95% Wald interval is narrowly above zero; alternative methods and design effects may change that conclusion. With this approximation, but the plausible range runs from almost no effect to a very strong one. A single iROAS of 1.35 would be misleading. The right conclusion is "ads probably work, and we don't yet know how well," and the next step is a larger or longer test before making a big budget decision.

From result to decision

Break-even iROAS depends on your margin. If your contribution margin before ad spend is 30%, an iROAS of 1.0 means you lose money on each sale, so you need an iROAS above 1 ÷ 0.30, about 3.3, to cover costs on first purchases (ignoring repeat purchases and lifetime value).

What If You Have a Small Budget?

You probably can't run a statistically perfect test, but you can still reduce guesswork:

1. Use a geo or time test on your biggest, most expensive campaign, and accept wide uncertainty.
2. Test the highest-risk spend first: brand terms, retargeting and always-on campaigns aimed at people already close to buying.
3. Run the test longer instead of bigger. More weeks add conversions.
4. Prefer a larger expected effect. Test a large change (turn a campaign off) rather than a small one (change a bid by 5%).
5. Triangulate. Combine the test with attribution trends, search-term analysis and sales data from your own records.
6. Repeat it. Two or three imperfect tests that agree are more convincing than one.

The Hidden Contaminant: Invalid Traffic

Competitor guides cover contamination from audience overlap and spillover. They rarely mention a third source that matters directly to incrementality: clicks and visits that aren't from real, interested people.

Bots, click farms and other invalid traffic can distort a test in several ways:

• They inflate spend with no possible lift. Fake clicks cost money but can never convert. In our worked example, if 15% of the $12,000 went to invalid clicks, a hypothetical iROAS excluding that spend would be about 1.59. Actual campaign iROAS remains 1.35 unless the cost is refunded; future prevention may improve returns but must be measured.
• They add noise to both groups, which widens confidence intervals and makes a real effect harder to detect.
• They can bias exposed and control groups differently, for example if bots hit the exposed audience's ads but not the control group, which can dilute measured conversion rates in the exposed group.
• They make attributed numbers look better or worse than they are, because fake clicks and visits show up in reported clicks, sessions and cost per click.
• They are more common on some placements and keywords, which can confuse comparisons across channels.

How to protect your test:

1. Clean the traffic before you measure. Detect and block invalid clicks first. See what click fraud is and our click fraud prevention guide.
2. Measure on conversions and revenue, not clicks or sessions, which fake traffic can inflate.
3. Check for odd patterns by source and region before the test starts, and exclude suspicious sources from both groups.
4. Compare your cost per click and spend against valid-click counts when interpreting iROAS.
5. Remember that fake traffic also ruins ordinary A/B tests. See mobile A/B testing and bot traffic.

Common Mistakes

• Using reported ROAS as proof that ads work
• Running a test too small to detect a realistic lift
• Changing budgets, creative or promotions during the test
• Comparing regions that aren't similar
• Stopping the test before late conversions arrive
• Testing during a major sale or launch
• Ignoring the confidence interval and acting on a single number
• Letting audiences overlap between exposed and holdout groups
• Measuring clicks instead of conversions
• Not checking for invalid traffic in either group
• Treating one test as permanent truth, when markets and costs change

Incrementality Test Checklist

• A clear question and a single outcome that matters
• A design chosen for my channel, volume and tools
• Required sample size or test duration estimated before launch
• Comparable test and control groups
• Budgets, creative, promotions and tracking frozen during the test
• Test long enough to include conversion lag and a full buying cycle
• Invalid traffic checked and blocked before and during the test
• Results reported with a confidence interval, not one number
• iROAS compared with break-even for my margin
• A date set to retest

Frequently Asked Questions

What is incrementality in marketing?

Incrementality is the extra results, such as sales or leads, that were caused by your marketing and wouldn't have happened without it.

What is incrementality testing?

It's a controlled experiment that compares a group exposed to your ads with a similar group that isn't, to measure the lift your ads caused.

What is iROAS?

Incremental return on ad spend: the revenue generated by the incremental conversions divided by ad spend. It differs from standard ROAS, which counts all attributed revenue.

How is incrementality different from attribution?

Attribution assigns credit to touchpoints for conversions that happened. Incrementality measures whether the ad caused those conversions.

How long should an incrementality test run?

Long enough to cover your normal buying cycle and conversion lag, often several weeks, and avoiding sales or launches. Longer tests add conversions and improve precision.

How big should the holdout group be?

It depends on your baseline conversion rate, the lift you want to detect and the precision you need. Some guides suggest 10% or more as a starting point, but calculate it for your own numbers.

Can small businesses measure incrementality?

Yes, in a rougher way: time-based or geo tests on large campaigns, longer durations, and combining tests with other data. Expect wide uncertainty.

What is a geo experiment?

A test that changes advertising in some regions and compares them to similar regions where nothing changed.

What is a conversion lift study?

A platform-run test where the ad platform randomly splits eligible users into an exposed group and a control group and reports the difference in conversions.

Can fake clicks affect incrementality tests?

Yes. Invalid clicks add spend and noise, can bias groups differently and make reported results unreliable. Block and monitor invalid traffic before and during tests.

Is a positive lift always a good result?

Not necessarily. The lift must be large enough to pay for the spend after your margin, and it must be reliable, not a noisy estimate.

Next Steps

Incrementality asks the question attribution can't: would this sale have happened anyway? Answer it with a controlled test, size the test before you launch, report the lift with a confidence interval, and compare iROAS with your break-even. Clean traffic matters too, because bots and click farms add spend and noise to both sides of the comparison. JuicyTraffic helps advertisers investigate paid-click quality and compare campaign evidence with qualified leads and sales.

References

• Google Ads: About conversion lift
• Google Research: Geo experiments

Related Articles

• Bidding on Branded Keywords: Is It Worth It? How to Measure the Real Return
• Marketing ROI: Formula, Worked Examples and How to Improve It
• How Invalid Traffic Distorts ROAS and Campaign Analytics

About the Author

Dylan Dan is the founder of JuicyTraffic. He has spent 15 years specializing in adult advertising and ad-fraud prevention, helping advertisers assess traffic quality, identify invalid clicks and protect media budgets across dedicated ad networks.