What 335,000 AI Crawler Visits in 30 Days Taught Us

TL;DR. One brand logged roughly 335,000 AI crawler visits in 30 days: about 118,000 from Meta-ExternalAgent, around 70,000 each from ClaudeBot and GPTBot, and roughly 50,000 from Amazonbot. None of it showed up in JavaScript tracking, and none of it is evidence of a citation. Crawling is an input, not an outcome.

We built AI crawler tracking to answer a question we could not answer about ourselves: does anything actually read our pages before an AI assistant answers a question about us? One busy brand on the platform logged roughly 335,000 crawler visits in a single 30-day window. None of it appeared in any dashboard the team already had, and the first conclusion we drew from the number turned out to be the wrong one.

What did 335,000 AI crawler visits in 30 days actually look like?

One brand, one domain, one 30-day window: roughly 335,000 AI crawler visits. Meta-ExternalAgent took about 118,000 of them. ClaudeBot took around 70,000. GPTBot took around 70,000. Amazonbot took around 50,000. Everything else spread thinly across a long tail.

We track 28 AI crawlers in total, and the shape of that distribution is the first thing worth internalising. Four bots carried the overwhelming majority of the traffic, and the remaining two dozen combined for a fraction of the total. If you had asked us to guess the ordering before we shipped the feature, we would have got it wrong. Most people do.

The second thing worth internalising is how quiet it all is. This is a normal mid-size site. No launch, no viral post, no press cycle. Just a domain sitting there, being fetched roughly eleven thousand times a day by machines that never appear in a session count, never trigger a conversion event, and never show up in any conversation about traffic. The team running that brand had been in weekly meetings for a year debating whether AI systems could see them at all.

Why do AI crawlers stay invisible to JavaScript tracking?

Because they do not run JavaScript. Page tags fire from a script executed by a browser, and a crawler that requests raw HTML and walks away never executes it. All 335,000 of those visits happened without producing one client-side event, which is why the team had no idea.

This is not a bug in anyone's setup and it is not something you can configure your way out of. It is a structural blind spot. The measurement layer most teams rely on was designed for humans holding mice, and the population it cannot see is now the population that decides whether you appear in an AI answer. You can have a perfectly instrumented site and still be completely blind to the traffic that matters most for citations.

The only place the truth lives is the edge. That is why our AI crawler tracking reads request logs rather than page tags: a small Cloudflare Worker deployed on your existing zone, about 5 minutes of setup, and the Cloudflare free plan is enough to run it. There is nothing clever about the mechanism. The clever part is looking at all, because almost nobody does.

Which AI crawlers should you expect to see?

Expect a leaderboard that does not match your intuition. In our 30-day sample the largest single source was Meta-ExternalAgent, not GPTBot. ClaudeBot and GPTBot landed in a near tie behind it, and Amazonbot came fourth with roughly 50,000 visits.

The instinct when you first see this is to feel flattered by the big numbers, and that instinct needs killing early. Crawl volume is mostly a function of the crawler's own appetite and refresh strategy, not a measure of how interesting you are. Some bots re-fetch aggressively by design. Some sample lightly and rely on other sources. A bot hitting you a hundred thousand times has told you about its scheduling policy, not about your standing.

What the per-bot split does tell you is coverage. Different crawlers feed different destinations, and being fetched constantly by one while never being touched by another is a real, actionable asymmetry. If ClaudeBot and GPTBot both read you every day and PerplexityBot has not requested a page in three weeks, you have a specific gap with a specific address, and that is far more useful than a headline total.

The other genuinely useful view is per-page rather than per-bot. We report recrawl cadence for every URL, and it is routinely uncomfortable reading. Teams discover that their pricing page, the one they rewrote two months ago, has been fetched twice since. Every AI answer describing their pricing is describing the old page. That is a fixable problem you can only see if you are looking at the log.

Does more AI crawling mean more AI citations?

No, and this is the mistake we care most about. Crawling is an input and citation is an outcome. A brand can be fetched 335,000 times in 30 days and still be named in almost none of the answers its buyers actually see. We have watched precisely that happen.

The temptation is obvious. Crawl volume is a big satisfying number that goes up, and it arrives without any of the ambiguity that surrounds real visibility measurement. It looks like progress. Teams put it on a slide. We very nearly built the product that encourages that, and we pulled back specifically because we could see how easily a large crawl count would become a substitute for asking whether anything came of it.

The useful framing is a pipeline with two ends. Crawling tells you raw material is entering. Citation tells you something finished came out. Confusing the two is how you end up celebrating a busy factory that ships nothing. So we treat the crawler view as a diagnostic that is only ever read alongside your AI Visibility Score, never on its own, and the interesting cases are the two mismatches.

What does heavy crawling with no citations tell you?

It tells you the technical layer is fine and your problem is content and authority. If four major crawlers fetched your pages hundreds of thousands of times and you are still absent from answers, discoverability is solved. The bots found you and decided what they read was not worth repeating.

This is the more common of the two mismatches and the harder one to accept, because it removes the comfortable explanation. Nobody is blocking you. Your sitemap is fine. Your schema is fine. The machines came, they read everything, and they chose someone else. That is not a crawling problem you can engineer away over a weekend.

Where we send teams in this situation is the four AI Visibility pillars, in order: how often you are named at all, how often you are actively recommended rather than merely listed, how many distinct high-authority sources are cited about you, and only then how positively you are described. Heavy crawl with no mentions almost always resolves to the third one. The crawlers can see what you say about yourself, and nobody else is saying it. Corroboration is slow, it is earned outside your own domain, and no amount of additional publishing on your own site substitutes for it.

What does light crawling with no citations tell you?

The opposite diagnosis, and a much cheaper fix. If a bot barely fetches you, nothing downstream can possibly work. Check crawler permissions, blocked user agents at the edge, an over-broad firewall rule, a stale sitemap, and pages no internal link points at.

We see accidental blocking far more often than deliberate blocking. A security rule written to stop scrapers catches GPTBot as collateral. A bot-fight setting gets enabled during an incident and never gets reviewed. A robots.txt wildcard written five years ago now excludes crawlers that did not exist when it was written. In every one of those cases the team believes their site is open, and the log says otherwise within about a minute of looking.

The coverage gap view is built for this. It lists the pages you care about that AI crawlers have never fetched, which is a different and more actionable question than whether your domain is crawled overall. A domain can be heavily crawled at the homepage and completely unread three clicks deep, and the pages three clicks deep are usually the ones carrying the specific claims you want repeated in an answer.

How do you read crawler data without fooling yourself?

Compare shapes, not totals, across the four windows we provide: today, 7 days, 30 days and 90 days. A one-day spike is a crawler changing its own schedule. A sustained shift on the specific pages you changed, holding for more than a week, is a signal.

We are strict about this internally because crawler data is unusually easy to over-read. It is high volume, it is noisy, and it moves for reasons that have nothing to do with you. Any single-day reading is meaningless. Any comparison that does not hold the page set constant is meaningless. And any improvement claimed before a week has passed is a coin flip dressed as a result, which is why nothing in our outcome tracking is judged before 7 days have elapsed.

The discipline that makes crawler data honest is pairing it with a dated record of what you changed. When you mark a finding as fixed, we compare the days after the fix against the same number of days before it, on the same page, across visits, AI crawler fetches, citations and indexing. That matched comparison is the only version of this data we trust, and it is the version that occasionally tells us a fix did nothing. Those are the readings worth having.

What is the bottom line on AI crawler traffic?

Start looking, then refuse to celebrate the total. Roughly 335,000 visits in 30 days across 28 tracked crawlers proves the pipe is open for one brand. It proves nothing about whether that brand gets recommended, and treating the two as one number is how teams waste a quarter.

The practical sequence is short. Turn on edge-level crawler tracking so you can see the population your page tags cannot. Read the per-bot split for coverage gaps rather than for reassurance. Read the per-page recrawl cadence to find the important pages going stale. Then put the whole thing next to your actual mention and citation data and let the mismatch tell you which problem you have. Heavy crawl and no citations is an authority problem. Light crawl and no citations is a plumbing problem. They need completely different quarters of work, and guessing which one you have is expensive.

We publish our own numbers on this because the alternative is a category full of vendors selling crawl counts as visibility. Crawl volume is the easiest number in this field to produce and the least meaningful one to act on alone. It earns its place only when it is read as an input to something you are measuring at the other end.

Frequently asked questions

How many AI crawler visits can one brand receive in 30 days?

One busy brand tracked by DataEase AI logged roughly 335,000 AI crawler visits in a single 30-day window. Meta-ExternalAgent accounted for about 118,000 of them, ClaudeBot for about 70,000, GPTBot for about 70,000, and Amazonbot for about 50,000. The rest spread across a long tail of smaller bots. DataEase AI tracks 28 AI crawlers in total, with today, 7-day, 30-day and 90-day windows.

Why does AI crawler traffic not appear in standard page tracking?

AI crawlers do not execute JavaScript. Page tags fire from a script that runs in a browser, and a crawler that fetches raw HTML never runs it. Every one of those 335,000 visits happened without producing a single client-side event, which is why most teams have no idea it is happening. The only reliable place to see it is the edge or the server log.

Does more AI crawling mean more AI citations?

No. Crawling is an input and citation is an outcome. A brand can be fetched hundreds of thousands of times in a month and still be named in almost none of the answers its buyers see. Reading crawl volume as citation volume is the single biggest mistake teams make when they first see crawler data.

What does heavy AI crawling with no citations mean?

It means discoverability is solved and the problem is content and authority. If four major crawlers fetched your pages hundreds of thousands of times and you are still absent from AI answers, the bots found you and decided what they read was not worth repeating. The fix lives in the four AI Visibility pillars: mentions, recommendation, citations and sentiment.

What does light AI crawling with no citations mean?

It means a technical discoverability problem. Check crawler permissions in robots.txt, blocked user agents at the edge, an over-broad firewall rule, a stale sitemap, and orphan pages no internal link points at. Nothing downstream can work if the fetch never happens, so this is always the first thing to fix.

How do you start tracking AI crawler traffic?

DataEase AI reads crawler traffic through a small Cloudflare Worker deployed on your existing zone. Setup takes about 5 minutes and the Cloudflare free plan is enough. Once it is running you get visits by bot over time, recrawl cadence per page, and a coverage gap list of important pages AI crawlers never fetch.