The blog · GTM Systems · From The Demand Compass · 6 min
Why do you need a holdout group to know if lead scoring works?
A score you act on cannot prove itself. A randomized 5 to 10 percent holdout is the only way to tell prediction from your own effort.

You need a holdout group because a score you act on can never prove itself. When a model flags an account as hot, a rep works it hard, and the deal closes, the score gets the credit even if the effort did all the work. A holdout is a small randomized slice of high-scoring accounts, five to ten percent, that the system does not route and that is handled as a control. The gap in outcomes between worked and held-out accounts is the only clean read on what the score is worth.
What is a holdout group in lead scoring?
A holdout group in lead scoring is a randomly chosen 5 to 10 percent of the accounts the model qualifies, deliberately left unrouted so that no rep acts on them, and compared against the accounts that were worked. It is a control group in the plain sense. Everything else about the two sets is the same, so any difference in outcome belongs to the thing you changed: whether the score was acted on.
We run one ourselves, and we describe it in the book as the single most important number in the chapter on results. It is not the flattering one. It is the one that keeps every other number honest. Source: automate rev.ops. deployments, The Demand Compass, chapters 17 and 18
Why can't a score prove itself without one?
A lead score cannot prove itself without a holdout because prediction and effort get tangled the moment anyone acts on the score. You score an account high, you work it hard because the score told you to, it closes, and you take credit for the score. That is a self-fulfilling prophecy, not a measurement, and from the inside it looks exactly like a model that works.
A rep handed a hot account calls first, writes a better opener and follows up faster, and any account treated that way converts more often. If the only accounts you ever compare are the ones you treated, you are measuring your own effort and attributing it to the arithmetic. You can no longer tell skill from a story you told yourself.
Without a control group, you can't tell skill from circularity.
How big should a holdout be, and how do you pick it?
A holdout should be 5 to 10 percent of the accounts that cross your qualifying threshold, chosen at random, and set aside before any rep sees the list. Random matters more than size. If a person picks which accounts to hold back, they will hold back the ones that look weaker, and the comparison is rigged before it starts.
In practice the system does the draw. The held-out accounts keep their scores, their signals and their timelines, so you can see later what happened to them without anyone having touched them. The slice is small enough that it costs a few accounts a quarter.
How do you read the result?
You read a holdout by comparing the conversion of worked accounts against held-out accounts over the same period, to the same outcome. If the worked accounts convert meaningfully better, the score is predicting something real. If they convert about the same, the score is fooling you, and the holdout says so in numbers you cannot argue with.
That second answer is useful, not a failure. A score that does no better than random is cheaper to fix when you know, and a holdout that confirms the score is a number you can put in front of a skeptical CFO. We do not assume our false-positive rate is low. We measure it.
The honest target is not zero. A zero false-positive rate means the threshold is so high that most of the real windows are missed too, so the aim is a rate low enough that reps still trust the queue, watched over time, because a creeping rate is the earliest sign the signals have gone stale. Source: automate rev.ops. deployments, The Demand Compass, chapters 18 and 20
| If worked accounts convert... | It means | What you do |
|---|---|---|
| Meaningfully better than held-out | The score predicts something real | Keep the weights, keep measuring |
| About the same as held-out | The score is not adding anything the effort did not | Recalibrate the weights and the decay |
What should you measure it against?
You should measure a holdout against whether the account became a real sales opportunity, a qualified pipeline event, not against closed-won. Closed-won is slow and noisy: it is contaminated by which rep worked the deal, how strong the sales team is, and plain luck. A qualified opportunity happens sooner and reflects what the score is actually trying to predict.
One rule keeps the labels honest. Never learn from a signal recorded after the outcome it is supposed to predict. A signal dated after the deal was already in motion is not predicting anything; it is reading the answer off the back of the test. A model trained on leaked answers looks brilliant in testing and fails on a fresh account. This is also where the decay rule earns its keep: the score should reflect what was true on the day it fired, not what an account looks like in hindsight.
Why do teams raid the holdout, and how do you stop it?
Teams raid the holdout because it looks like pipeline left on the table, and the first tight quarter makes that look unaffordable. Someone points at the accounts nobody is working, the accounts the score says are hot, and asks why they are sitting idle. It is a reasonable question and the wrong answer is to release them.
The day someone decides the holdout costs too much pipeline to maintain is the day you lose the ability to prove the whole system works. Protect it the way you protect a finance control: write down, while it is calm, that nobody may release held-out accounts, and name who is allowed to change that rule. The cost is a handful of accounts. The benefit is the only instrument that tells you whether the other ninety percent of your effort is aimed well.
Related questions
Isn't a holdout just leaving money on the table?
For the few accounts in the slice, yes, and that is the price of knowing. Five to ten percent of qualifying accounts is small, and the alternative is a score nobody can verify, which costs far more when it quietly stops working.
Does a holdout replace an A/B test of messaging?
No. A holdout tests whether acting on the score beats not acting. A message test compares two ways of acting. You can run both, but the holdout comes first, because there is no point optimizing the copy to accounts the score should not have picked.
What if our volume is too low for a statistically clean holdout?
Then read it as direction, not proof, and let it accumulate over more quarters. A small control that runs for a year still beats no control, and the timelines of held-out accounts are worth reading one at a time even before the totals mean anything.
Who is allowed to change the weights once the holdout shows a gap?
A human, on evidence. The system can study outcomes and propose new weights offline, run them in shadow alongside the live model, and show the lift and the sample size. Only a person promotes the new version, and the old one stays in place so a rollback is a status flip.

