Every request to your site leaves a line in the web server or CDN access log, and that line carries two fields that matter here: the user agent string and the source IP address. AI traffic is found by filtering on the first and confirmed with the second. The user agent tells you which bot claims to be visiting. The IP address tells you whether the claim is true. Done together, the two checks separate genuine AI crawlers from impersonators, and separate training crawls from the visits that happen when a person asks an assistant a question.
Why do server logs show AI traffic that analytics does not?
Analytics tools such as Google Analytics measure visits by running JavaScript in a browser. Most AI crawlers and fetchers request the raw HTML and do not run that script, so they never trigger a tag and never appear in a report. The web server still serves the page and still writes a log line, which makes the access log the most complete first-party record of AI activity on a site.
This is also why log data and analytics data disagree. If you want the longer explanation of the gap, see why AI bots do not show up in Google Analytics.
Where do the logs live?
The location depends on how the site is served.
- Origin web server: Apache and Nginx write access logs to disk, commonly under
/var/log/. These only contain requests that reached the origin. - CDN or edge provider: Cloudflare, Fastly, Akamai and CloudFront can export request logs. These are usually the better source, because cached responses never reach the origin and so never appear in origin logs.
- Managed hosting: many platforms expose logs through a dashboard or an export. Check whether the export includes the user agent field, as some summarise it away.
If you only have origin logs, treat every count as a floor rather than a total.
Which user agent strings identify the main AI bots?
Match on the bot token rather than the whole string, because vendors change version numbers. The tokens below come from each vendor's own documentation.
OpenAI publishes three bots at developers.openai.com. GPTBot crawls content that may be used to train its models. OAI-SearchBot surfaces sites in ChatGPT search features. ChatGPT-User visits pages when a person asks ChatGPT or a custom GPT a question. OpenAI also lists OAI-AdsBot, which checks landing pages submitted as ChatGPT ads.
Anthropic documents three bots on its support site. ClaudeBot collects content that may be used for training. Claude-SearchBot browses to improve search result quality. Claude-User retrieves pages when an individual asks Claude a question.
Perplexity declares PerplexityBot and Perplexity-User, as described in Cloudflare's August 2025 investigation. Google is the awkward case. Google-Extended is a robots.txt control token only. Google states it has no separate HTTP user agent string, so it will never appear in your logs, and Google's AI-related crawling arrives under existing Google user agents. For a fuller reference, see the AI crawler and bot user agent list.
What is the quickest way to count AI bot requests?
On a standard combined-format access log, one command pulls the matching tokens and counts them:
grep -Eio 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log | sort | uniq -c | sort -rn
To see which pages they request, print the request path for the same matches:
grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -25
The second view is often the more useful one. It shows which articles the assistants fetch, which is a rough proxy for the content they consider worth retrieving.
How do you tell training crawls from live answer retrieval?
The tokens map to different purposes, and the purpose changes what the visit means for you. Training crawlers such as GPTBot and ClaudeBot gather content for model development and tend to sweep broadly across the site. Search bots such as OAI-SearchBot and Claude-SearchBot build the index behind an assistant's search feature. User-triggered fetchers such as ChatGPT-User and Claude-User fire because a person asked a question, so each hit is closer to a reader's moment of intent than to background collection.
Split your counts into those three groups. A site with heavy training volume and little user-triggered volume is being harvested. A site with steady user-triggered hits is being used to answer questions, which is the traffic with a plausible case for monetisation. The distinction is explained further in what is a live search agent.
How do you confirm a bot is who it says it is?
A user agent string is text the client chooses, so anyone can send it. Verification means checking the source IP.
For OpenAI, compare the IP against the JSON ranges the company publishes for each bot, at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. Anthropic publishes its crawler addresses at claude.com/crawling/bots.json, and states that a crawler whose source IP is on that list is identified as coming from Anthropic.
Google documents a two-step DNS method. Run a reverse DNS lookup on the IP, confirm the domain is googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP. Google also publishes its ranges as JSON files for automated matching.
Requests that claim an AI user agent but come from an address outside the published range are not that vendor. Flag them separately and do not count them in your AI totals.
What if the traffic does not announce itself?
Some AI traffic uses a user agent that looks like an ordinary browser. In August 2025 Cloudflare reported that when Perplexity's declared crawlers were blocked, requests continued from a generic Chrome-on-macOS user agent, rotating IP addresses and networks outside Perplexity's published range. Cloudflare estimated roughly 20 to 25 million daily requests from the declared crawler and 3 to 6 million from the undeclared one, across tens of thousands of domains. Perplexity has disputed how that traffic should be characterised, so treat the figures as Cloudflare's measurement, not a settled fact.
The practical lesson is that a log filter on named tokens finds the bots that identify themselves and misses the ones that do not. Undeclared traffic shows up through behaviour: very high request rates from one network, no asset requests such as images or stylesheets, no cookies, and systematic page-by-page patterns. Bot management products score these signals, and a CDN-level view is usually where they are caught. See also what is a stealth crawler.
What should you do with the numbers?
Start with a baseline: requests per bot group per week, the top fetched pages, and the share that fails IP verification. Compare it against your analytics sessions to size the gap. Then decide per group. Training crawlers are a licensing and robots.txt question. Search bots are a visibility question. User-triggered fetchers are a revenue question, because they are the visits where an answer is being assembled in front of a person.
Logs are a record after the fact. Detecting a user-triggered fetch while it happens, at the CDN edge, is what makes it possible to act on it, for example by serving a contextual brand mention into the response. That is the layer blankspace works at. For the broader decision, see should publishers block or monetise AI crawlers.
Frequently asked questions
Can Google Analytics show AI crawler traffic?
Not reliably. Google Analytics depends on JavaScript running in a browser, and most AI crawlers and fetchers request the HTML without running it. Server or CDN logs record every request that reaches them and are the better source.
Will I see Google-Extended in my logs?
No. Google states that Google-Extended has no separate HTTP user agent string and works only as a robots.txt control token. Training and grounding controls are applied through that token, but the requests themselves arrive under existing Google user agents.
How do I know a request really came from OpenAI or Anthropic?
Check the source IP against the ranges the vendor publishes. OpenAI provides separate JSON files for GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic provides one at claude.com/crawling/bots.json. A matching user agent from an address outside those ranges is an impersonator.
What is the difference between ChatGPT-User and GPTBot?
GPTBot crawls content that may be used to train OpenAI's models. ChatGPT-User visits a page because a person asked ChatGPT or a custom GPT a question. OpenAI notes that robots.txt rules may not apply to ChatGPT-User, since the visit is user-initiated.
Do I need a CDN log to do this properly?
It helps a great deal. Cached responses are served by the CDN without reaching your origin, so origin logs undercount. A CDN log captures requests before caching, and it is also where bot scoring and verification are usually available.