Why We Deleted Our Predicted Score Lift

TL;DR. Every finding used to carry a predicted score lift. We found 2 estimators in the codebase, neither ever compared against a real outcome, and deleted both. Findings now show impact and effort, and we measure the fix afterwards as a matched pair with a 7 day floor.

For a while, every recommendation in our product came with a small confident number attached. Fix this and your score goes up by about eight. Customers loved it. It was completely made up, we had built it twice, and we deleted both versions. This is the confession and what we learned from it.

What was the predicted score lift and why did we delete it?

It was a number rendered next to every finding, promising roughly how many points your score would rise if you fixed it. We deleted it because it was invented. Two separate estimators existed in the codebase and neither had ever been compared against a real outcome.

The feature felt indispensable while we had it. Founders open a list of thirty findings and immediately want to know which one is worth their Tuesday. A predicted lift answers that instantly and satisfyingly. It sorts the list, it justifies the effort, and it turns a vague chore into something that looks like a plan. Nobody ever asked us to remove it.

What broke the spell was a support conversation. A customer had done a fix we had priced at a large predicted gain, waited, and seen nothing move. They asked, reasonably, what had gone wrong. We went to look at how the estimate had been produced so we could explain the gap, and discovered there was nothing to explain. There was no model. There was no history. There was a mapping.

How does a fabricated number get built by accident?

Gradually and with good intentions, which is the uncomfortable part. Neither estimator was written by someone trying to mislead anyone. Both started as a placeholder to unblock a screen, and a placeholder that ships becomes a feature that nobody revisits.

The mechanism was almost embarrassingly simple. A finding carries a priority label. The estimator took that label, looked up a number, and returned it. High priority produced a bigger number than medium, medium produced a bigger number than low. That is the entire algorithm. Dressed in a progress bar and a plus sign, it read as a forecast produced by a system that knew something. What it actually encoded was the opinion already contained in the word "high", laundered into arithmetic.

Then it happened a second time. A different surface needed the same idea, the original function was not where anyone looked, and a second estimator appeared with slightly different constants. So the product could show two different predicted lifts for the same class of fix depending on which screen you were on. We only found that because we went looking after the support ticket. This is how it always goes: the fake number is not a decision anyone made, it is a shortcut nobody reviewed, duplicated by a team moving fast.

The thing worth sitting with is that the numbers looked more credible than everything around them. Precision reads as rigour. A prediction of "+8" is more persuasive than a label saying "high impact", even though the label is the only honest thing in the pair, because the label admits it is a judgement and the number pretends it is not.

What was actually wrong with the estimate?

It made a claim about the future that nothing in the system could support. Score movement depends on competitor behaviour, on how AI assistants sample your category that week, and on corroboration you do not control. None of that was in the lookup, because nothing was in the lookup.

Even a genuinely good estimator would have been hard here. To forecast a score change you need a population of comparable fixes, measured outcomes for them, and enough stability in the underlying environment for the past to say anything about the future. We had none of the first two and the third is genuinely doubtful in this category. That is not a reason to be sloppy. It is a reason to not make the promise.

There was also a subtler harm we underrated. The predicted lift did not just misinform, it reordered work. Founders sorted their list by the fake number and did the top items first. So a lookup table with no evidence behind it was quietly setting the roadmap for real companies. When we framed it that way internally, the argument for keeping it collapsed in about a minute.

What replaced the predicted lift?

Two honest signals and one real measurement. Every finding now shows impact and effort, which are judgements we can defend as judgements. And instead of promising an outcome in advance, we measure what actually happened after you ship the fix.

Impact and effort are not disguised as predictions. Impact is our assessment of how much this class of problem tends to matter, stated as a judgement. Effort is our assessment of what it will take you to do it, stated as a judgement. Two labelled opinions beat one fake number, because a reader can disagree with an opinion. Nobody argues with a decimal.

The genuinely useful replacement, though, is the measurement half. Findings live in a filterable list. When you mark one as fixed, it moves to a Tracking tab and becomes something we watch rather than something we predicted. Re-auditing a single page costs 1 credit instead of re-running the full 40-page crawl, and it comes back in roughly 6 to 12 seconds. That price was set deliberately. Verification has to be cheap enough that nobody skips it, because the entire point of deleting the forecast was to move the effort from guessing to checking. We wrote up the wider approach in our guide to proving AI visibility work actually pays off.

How do we measure whether a fix worked?

As a matched pair. We compare the N days since the fix against the N days immediately before it, on the same page, across visits, visits from AI, AI bot crawls, citations, indexing and page score. Nothing is judged before 7 days have passed.

The matched pair matters because the naive alternative is worthless. Comparing this month to last month conflates your fix with everything else that changed, including seasonality, a competitor launch, and a crawler adjusting its own schedule. Anchoring both windows to the fix date and keeping them the same length removes most of that, and it makes the comparison something you can look at and reason about rather than a number you have to take on faith.

The 7-day floor exists because we watched ourselves get excited too early. Two days after a change, everything looks like it moved, because crawler traffic is spiky and citation sampling is noisy. A one-day reading in this domain is a coin flip with a chart around it. So the product simply refuses to render a verdict before the week is up. It is not showing a weak result, it is declining to show one, and those are different.

The uncomfortable consequence is that the honest system sometimes tells you a fix did nothing. That happens, and we made a decision to show it rather than round it toward encouragement. Those readings are the most valuable output the tool produces, because they are the only ones capable of changing what you do next.

Why does no data render as a dash and never a zero?

Because a zero is a claim and a dash is an admission. Showing zero says we measured and found nothing. Showing a dash says we do not know yet. Those are different statements about the world, and conflating them is the same sin as the predicted lift in miniature.

This turned into a rule we enforce everywhere, and it catches more bugs than you would expect. A page with no crawl history yet is not a page with no crawls. A fix inside its first week is not a fix with no effect. A citation count that has not been refreshed is not a citation count of zero. In every one of those cases the default rendering of an empty variable would have produced a confident, wrong, discouraging number.

It also protects against the failure mode that started all of this. Fake numbers rarely enter a product through a big design decision. They enter through a default value, a placeholder, a null coalescing to zero. If your rule is that unknown must look unknown, most of that class of bug becomes visible in review instead of shipping quietly.

What should other teams take from this?

One rule: if a number in your interface cannot be traced back to something you measured, delete it. Not soften it, not add a tooltip, delete it. A fabricated figure is worse than a blank space because it looks like evidence and cannot be argued with.

The generalisation goes beyond forecasts. Any time a product presents a judgement as a measurement, it borrows credibility it has not earned, and the loan gets called in the first time a customer checks. We would much rather ship a screen that says "high impact, low effort, we will measure the result in a week" than one that says "+8" and cannot survive a follow-up question.

Three things we now do by default. Label judgements as judgements, in plain words, so nobody mistakes an opinion for an output. Spend the engineering effort on the after rather than the before, because measuring what happened is tractable and predicting what will happen usually is not. And treat every confident number in a review as guilty until someone can point at where it came from, because the two estimators we deleted both passed review the first time.

Deleting a feature that customers liked, to replace it with less certainty, is not a fun product decision. It is also the only version of this product we would be willing to defend to the person who asked us why their fix did not work.

Frequently asked questions

Why did DataEase AI remove predicted score lift from recommendations?

Because the prediction was invented. Two separate estimators were built, and both turned out to be a lookup that mapped a priority label to a number and printed it as a forecast. Neither had ever been checked against a real outcome. There is now deliberately no predicted score lift anywhere in DataEase AI.

What replaced predicted score lift?

Two honest signals plus real measurement. Every finding now shows impact and effort, which are judgements we can defend, and outcomes are measured after the fact rather than promised in advance. Findings live in a filterable table, and marking one as fixed moves it to a Tracking tab where the result is measured.

How does DataEase AI measure whether a fix worked?

As a matched pair. The N days since the fix are compared against the N days immediately before it, across visits, visits from AI, AI bot crawls, citations, indexing and page score. Nothing is judged before 7 days have elapsed, because anything shorter is noise wearing the costume of a result.

Why must missing data render as a dash rather than a zero?

Because a zero is a claim and a dash is an admission. Rendering absent data as zero tells the user a measurement was taken and came back empty, which is a different and more damaging statement than saying nothing is known yet. In DataEase AI, no data renders as a dash and never as a zero.

How much does it cost to re-check a single fixed page?

Re-auditing a single page costs 1 credit rather than re-running the full 40-page crawl, and it returns in roughly 6 to 12 seconds. That price was deliberate: verification has to be cheap enough that nobody skips it, because the whole point of deleting the prediction was to make people check the outcome instead.

What is the lesson for other teams shipping AI recommendations?

If a number in your interface cannot be traced to something you measured, delete it. A fabricated forecast is worse than a blank space because it looks like evidence and cannot be argued with. Ship the judgements you can defend, label them as judgements, and put your engineering effort into measuring what happened afterwards.