AI crawlers
The bots that crawl your site for AI, and why we don't track them in the plugin.
Separate from the people AI sends you are the bots that AI companies run to crawl the web. They visit your site too, but they are a different kind of signal, and Visibility Kit does not track them. This page explains what they are and where to look if you want to see them.
The three kinds of AI crawler
AI crawlers fall into three groups, based on what they are doing when they hit your page.
The first group fetches a page because a person just asked an AI about it. When someone asks ChatGPT or Claude a question and the tool needs to read your page to answer, it sends a bot to fetch it right then. These carry user-facing names like ChatGPT-User, Claude-User, and Perplexity-User. Of the three groups, this is the one closest to a real person, since a human is waiting on the other end.
The second group indexes your site so the tool can find it later. These are the search crawlers, like OAI-SearchBot, PerplexityBot, and Google's GoogleOther. They are not tied to a question in the moment, but their visits are what make it possible for a tool to cite you down the line.
The third group collects content to train future models. GPTBot, ClaudeBot, Google-Extended, and CCBot are the common ones. Site owners tend to have opinions about these, and you can allow or block them in your robots.txt.
Two names that look alike
A couple of these are easy to mix up, because the company is the same but the activity is not. ClaudeBot is Anthropic bulk-crawling your site to train future models. Claude-User is a one-off fetch because a specific person just asked Claude about your page. GPTBot and ChatGPT-User split the same way: one is training, the other is a live retrieval for someone who is waiting. When you read your logs, that distinction is what tells you whether you are looking at background scraping or real-time interest.
Why Visibility Kit doesn't track these
Crawlers do not run JavaScript. The script that captures your AI referral traffic runs in a real browser, and a bot is not a browser, so a tool like Visibility Kit never sees these visits. The only places that do see them are your server logs and your CDN.
Earlier versions of the plugin detected these bots on your server and reported them back to us. We removed that in 2026. The honest reason is that the counts did not earn their keep. Knowing that a training bot crawled you two hundred times last week did not change what anyone did, while a person arriving from an AI answer and filling out a form does. We would rather measure the thing that maps to a real outcome.
If you do want to watch crawler activity, your CDN already records it, and that is the right place to look. The Cloudflare page covers how. For a fuller reference on the individual bots, we keep a running list on the Complete SEO blog: the complete list of AI crawl bots and what they do.