AI lead scoring: when it beats a points system, and how to test it before reps trust it

This guide is for marketing and sales operations leads, sales managers and founders deciding whether to replace or supplement a points-based lead score with AI. It covers how predictive scoring works, how much data you need, how to judge a model, the mistakes that make a model look better than it is, and a backtest you can run. If you don't have a scoring model yet, start with the free lead scoring template, which builds a points-based score you can later test an AI model against.

Points-based or AI lead scoring?

Most teams that succeed with AI scoring run both for a while: fit rules to keep out leads you'd never sell to, and a predictive score to rank the rest.

How AI lead scoring works

1. Collect closed leads. Each one is labeled converted (qualified, became an opportunity or a customer) or not.
2. Describe each lead with features. Fit data (industry, size, location, role) and behavior data (pages visited, forms filled, emails answered).
3. Train on older leads, test on newer ones. Microsoft's Dynamics 365, for example, trains on 80% of closed leads and validates on the most recent 20%.
4. Score open leads. The model outputs a probability, shown as a score, plus the top factors pushing it up or down.
5. Retrain on a schedule. Dynamics 365 offers automatic retraining every 15 days; whatever the tool, retrain when your market or offer changes.

Do you have enough data?

Vendors publish the minimum needed to build a model:

These are floors, not targets. With 40 conversions, a model can't tell a real pattern from a coincidence in more than a couple of features. As a working rule, if you have fewer than a few hundred conversions in a period when your market and offer stayed the same, a points system built from your best customers is safer, and you can test a model against it later.

Data quality matters as much as volume. If reps close leads as "disqualified" to tidy their queues, or "converted" means something different by region, the model learns your record-keeping habits.

How to judge a scoring model

Ignore accuracy on its own. Microsoft's documentation shows a sample model with 75% accuracy on a dataset where 70% of records converted. A model that predicted "yes" for every record would have scored 70%. Accuracy only means something when compared with the base rate.

Use AUC to compare models. The area under the curve is the probability that the model ranks a randomly chosen converted lead above a randomly chosen non-converted one. 0.5 is a coin toss; 1.0 is perfect. Dynamics 365 marks a model "not ready to publish" when its AUC falls below a threshold.

Use lift to decide whether it's worth acting on. Sort held-out leads by score, split them into ten equal groups (deciles), and compare each group's conversion rate with the average. If the top tenth converts at three times the average, reps who call that tenth first will get three times the conversions per call. Lift is the number sales leaders understand.

Check calibration if you use the number. If leads scored 80 convert 20% of the time, the score ranks well but isn't a probability. Use it for ordering, not for forecasting.

Mistakes that make a model look better than it is

• Leakage. Fields that are only filled in after a rep has worked a lead, such as "meeting booked", "assigned producer" or a status set during qualification, let the model see the answer. The backtest looks superb and live performance collapses. Train only on what you knew when the lead arrived.
• Feedback loops. Once reps call only high scores, high scores convert more because they were called, and the model's next training round confirms itself. A random holdout breaks the loop.
• Learning your past, not your market. A model trained on your existing customers scores look-alikes highly and new segments poorly. Score new markets with rules until they have history.
• Behavior signals that aren't people. Security scanners and privacy features can trigger email opens and clicks automatically. Weight replies and form fills over opens.
• Drift. A price change, a new product or a new channel changes who converts. Watch top-decile conversion monthly.
• A score nobody uses. If routing, alerts and rep views don't change, the score is decoration. Decide the action for each band before you launch.

Worked example: a backtest at a commercial insurance broker

Illustrative example. The broker and its numbers are invented to show the method.

The broker: Copperline Insurance Brokers (copperline.example) gets about 4,800 commercial quote requests a year for property, liability and fleet cover. A request "converts" when a policy is bound. It has a points-based score built two years ago.

The backtest: a predictive model is trained on requests from January 2025 to June 2026, then scores the 1,200 requests from July to September 2026, which it has never seen. 96 of them were bound, an 8% base rate.

First result, too good: AUC 0.91. The top factor was "producer assigned", a field the team fills in after the first call. With it removed, AUC fell to 0.78, which is the honest figure.

Lift after removing the leak:

The top 30% of requests by AI score held 63 of the 96 bound policies (66%). The same top 30% by the old points score held 47 (49%). If producers work the top 30% first, they reach 16 more eventual policies in the same number of calls.

The decision: requests in the top 30% go to a producer within an hour; the middle 20% get a call within two business days; the bottom half get an emailed quote form and a follow-up. A random 10% of all requests bypass the score and are worked in arrival order, so Copperline can check each quarter that top-decile conversion holds up when everyone gets the same attention.

A backtest you can run

1. Export closed leads for 12 to 24 months with the outcome and the date each was created.
2. Remove leaky fields: anything set after first contact.
3. Train on the older part, test on the most recent quarter. Never test on leads the model trained on.
4. Rank the test leads by the AI score and, separately, by your current score.
5. Compare lift in the top 10% and top 30%, and the share of all conversions each captures.
6. Read the top factors. If one looks like an outcome rather than a signal, you've found a leak.
7. Launch with a random holdout of 10%, and recheck lift every month for the first quarter.

What to ask a vendor

• What is the minimum data to build a model, and what do you recommend for a reliable one?
• Which fields does the model use, and can I exclude fields set after first contact?
• Can I see AUC and lift by decile on held-out recent leads, not just accuracy?
• How often does it retrain, and can I see performance change after each retrain?
• Does it show the top factors behind each score?
• How does it handle new segments with no history?

What it costs

Predictive scoring is usually bundled into a higher tier rather than sold alone. As checked on 26 October 2026, Microsoft lists Dynamics 365 Sales Enterprise at $105 a user a month paid yearly, with predictive lead scoring included (quick setup includes 1,500 scored records a month), and HubSpot requires Marketing Hub Enterprise for AI contact scores. Score the fit side first: Bebop explains why each company it finds fits your description, which gives a points system cleaner inputs from day one.

Next step: build your baseline with the lead scoring template, then backtest any AI model against it.