Did the Fix Actually Work?

TL;DR. We measure whether an AEO fix worked with a matched pair: the N days since the fix against the N days immediately before it, on the same page, across 6 measures. Nothing is judged before 7 days, missing data renders as a dash rather than a zero, and a single page re-audit costs 1 credit.

The hard question in answer engine optimization is not what to fix. Findings are easy to generate and most teams have more of them than they can ship. The hard question arrives three weeks later, when somebody asks whether the thing you fixed did anything, and the honest answer is usually a shrug wrapped in a chart.

How do you tell whether an AEO fix actually worked?

Compare the same length of time on both sides of the change. Mark the fix date, wait at least 7 days, then read the N days since against the N days immediately before, across the same 6 measures. Anything else is an anecdote with a date on it.

What most reporting does instead is show one line trending upward with a marker on it somewhere. The eye does the rest, and the eye is generous. A line that was already rising before the fix will look like it rose because of the fix, and nobody in the room is motivated to say otherwise. We have all sat in that meeting.

The mechanics in the product are deliberately mundane. Findings live in a filterable table. You ship something, you mark the finding as fixed, and it moves to a Tracking tab with a fix date attached. From that moment the page has a before and an after, and the comparison is computed rather than eyeballed. The value is not in the sophistication of the method, it is in the fact that the date got written down at all.

What is a matched pair, and why do we compare that way?

A matched pair is two windows of identical length, one on each side of the fix date. 14 days after against the 14 days immediately before, never 14 against 30. Equal windows keep the comparison about the change.

Unequal windows are the most common way to accidentally lie to yourself. Compare two weeks after against a month before and the earlier window contains more of everything, so the fix looks harmful. Flip it and the fix looks miraculous. Neither result says anything about your page. It says something about how much calendar each side of the comparison happened to contain, which is not a finding.

Crawl and visit patterns are also strongly weekly. Bots have cadences, buyers have working days, and a window that starts on a Wednesday and ends on a Tuesday carries a different mix of both than one that starts on a Monday. Multiples of 7 days on each side are worth more than a round number of days, which is another reason the 7 day floor is a floor rather than an arbitrary delay.

We should say what a matched pair is not. It is not a randomized experiment. There is no control page sitting untouched next to the one you changed, and there cannot be, because the pages are not interchangeable. What you get is a disciplined observation, not proof.

Why do we refuse to judge anything before 7 days?

Because AI crawlers do not return on your schedule. Recrawl cadence for a single page can be days apart across the 28 AI crawlers we track, so nothing is judged before 7 days, and the view says the window is too short rather than guessing.

The pull to check the next morning is very strong, and we have felt it. You ship a fix at midnight, you open the tracking tab at breakfast, and you want the machine to tell you something. But the first 48 hours after a change contain almost nothing except your own re-checks. The crawler that mattered has not been back yet. Whatever the screen shows you at that point is noise with a confident font.

The damaging version of this is not impatience, it is reversal. A founder ships a genuinely good change, checks on day two, sees a flat line, decides the change did not work and reverts it. Two days later the crawler arrives and fetches the reverted page. That is a fix that was never actually tested, and the conclusion drawn from it was the exact opposite of the truth. Refusing to render a verdict early is the cheapest way we found to prevent that.

Which six measures do we compare across the pair?

Six, shown side by side rather than rolled into one verdict: visits, visits from AI, AI bot crawls, citations, indexing and page score. They move on different clocks, and the order in which they move is itself the diagnosis.

• Page score. Immediate on re-audit. It is the only measure entirely under your control, and it tells you the change landed, not that it worked.
• AI bot crawls. Fastest external signal. Did anyone come back and look at the page after you changed it?
• Indexing. Whether the page is picked up and retained, which is a prerequisite for everything downstream.
• Citations. The slowest to move and the closest to what you actually wanted, since a citation means a source about you was worth naming.
• Visits and visits from AI. The business outcome, and the noisiest pair on the list. Read them last.

The sequencing is the part people miss. These are not six independent scores to average, they are a chain. If AI bot crawls did not move, nothing further down the chain is going to move either, and you have learned something useful in a week rather than a quarter: your change was never seen. If crawls rose and citations did not, the page was read and rejected, which is a content problem rather than a discoverability problem. Those two situations call for completely different next actions and look identical in a single blended number.

Why does "no data" render as a dash and never as a zero?

Because a zero is a claim and a dash is an admission. A zero next to a fix says nobody came. A dash says we do not know. Those are different facts, and only one of them is true when the measurement is simply missing.

This sounds like a pedantic display detail and it is the most consequential decision in the whole feature. Zeros are load-bearing. A column of zeros beside a change you were proud of reads as a verdict, and people act on verdicts. We have watched teams abandon a direction because of a zero that was really an absence of instrumentation, a crawler that had not visited yet, or a data source that had not been connected on the day in question.

So the rule is absolute: if we did not observe it, we do not render a number for it. Same family of decision as deleting our predicted score lift, which we wrote about in why we deleted our predicted score lift. A missing measurement and a measured absence are not the same thing, and a product that blurs them is training its users to trust it in exactly the moments it should not be trusted.

What happens when you change the fix date?

You can change it, and you should when it is wrong. Editing the fix date re-bases every window on both sides so the pair stays matched. The one thing that never recomputes is the original baseline snapshot taken when the finding was first marked fixed.

Editability is not a convenience feature, it is an accuracy feature. People mark things fixed days after they shipped them, because marking things is admin and shipping is work. Or the deploy went out Tuesday night and got marked on Thursday. If the date is wrong the pair is misaligned, and a misaligned pair quietly includes some of the after period inside the before window, which drags the result toward no effect. Letting you correct the date fixes a real measurement error.

Freezing the baseline snapshot is the counterweight, and it is deliberately a constraint on us as much as on you. If the baseline recomputed every time the date moved, you could nudge the date around until the comparison looked good, and never tell a single lie doing it. The snapshot is the record of the state of that page at the moment somebody declared it fixed. It is allowed to be inconvenient. That is what makes it evidence rather than decoration.

Why does re-auditing one page cost 1 credit instead of a full crawl?

Because re-running a 40 page crawl to check one page you changed is slow and wasteful. A single page re-audit costs 1 credit and returns in roughly 6 to 12 seconds, so verification happens in the same sitting as the fix rather than next month.

Friction determines behavior far more than intent does. When checking your work costs a full crawl and a wait, nobody checks their work. The fix ledger degrades into a to-do list of things somebody once did, with no record of whether any of them mattered, and the whole apparatus of tracking becomes theater. Bringing verification down to one credit and a few seconds is what makes the loop close.

The audit also meets you halfway. Self-fix detection notices when a page is new or visibly changed and is named by an open recommendation, then asks whether you fixed it, verifying up to 12 pages per run. It suggests and it never auto-applies, because a system that silently marks its own recommendations as complete is grading its own homework. Every scan shows its exact price before it runs, and credits are refunded when a run fails.

What can this method still not tell you?

It cannot prove causation. A matched pair is a disciplined observation with no control group, so it rules things out cheaply rather than proving things expensively. For a small team shipping weekly, that trade is almost always the right one.

The clearest limitation is bundling. Ship six changes in one week and the pair cannot separate them. You will know the week did something and you will not know which change. The practical answer is not to build a lab, it is to space the fixes you genuinely care about attributing and accept a week-level verdict on the rest. Perfect attribution is not worth the pace you would give up to get it.

The second limitation is that the outside world keeps moving. A competitor publishes, an assistant changes how it sources answers, your category has a busy month. None of that is in the pair, and all of it is in the numbers. This is exactly why we show 6 measures instead of one score: a broad external shift usually moves several of them together, and a real page-level fix usually moves the chain in order, starting with crawls.

If you want the longer version of how we think about evidence rather than assertion, it is written up under proving AI visibility work actually works. The short version is the one we keep coming back to internally: write down the date, wait the week, compare equal windows, and let a dash mean what a dash means.

Frequently asked questions

How do you tell whether an AEO fix actually worked?

Compare the same length of time on both sides of the change. DataEase AI marks the fix date, waits at least 7 days, then reads the N days since the fix against the N days immediately before it, across 6 measures. Anything shorter or unmatched is an anecdote with a date attached to it.

What is a matched pair in outcome tracking?

A matched pair is two windows of identical length, one on each side of the fix date: 14 days after against the 14 days immediately before, never 14 against 30. Equal windows keep the comparison about the change rather than about how much calendar each side happened to contain.

Why does DataEase AI refuse to judge a fix before 7 days?

Because AI crawlers do not return on your schedule. Recrawl cadence for a single page can be days apart across the 28 AI crawlers DataEase AI tracks, so nothing is judged before 7 days. The tracking view says the window is too short rather than showing an early verdict that would be wrong.

Why does no data show as a dash instead of a zero?

Because a zero is a claim and a dash is an admission. A zero next to a fix says nobody came and nothing happened. A dash says we do not know yet. Founders revert good changes over zeros that were really an absence of data, so DataEase AI never renders missing measurements as 0.

How much does it cost to re-audit a single page after a fix?

A single page re-audit costs 1 credit and returns in roughly 6 to 12 seconds, instead of re-running a 40 page crawl to check one change. Every scan shows its exact price before it runs, and credits are refunded when a run fails, so verification happens in the same sitting as the fix.