Last updated August 2026. User-agent strings confirmed against each vendor”s published documentation; GA4 hostname list updated to reflect Claude referral growth.
AI engines do not crawl like Googlebot. They send their own bots, on their own schedules, for their own indexing purposes. Those bots leave a clear trace in your server logs. You just need to know what to look for.
This piece walks through the full picture: how to identify AI crawler activity in raw log data, how to understand what it means for your citation coverage, and how to connect that upstream crawl data to the downstream referral sessions your AI visibility platform and GA4 should both be tracking.
Why server logs matter for AI visibility
Most brand monitoring focuses on the output side: run prompts against ChatGPT, Perplexity, or Gemini, and measure how often your brand appears. That is the right starting point.
But the output side only tells you what the model surfaces. It does not tell you which of your pages the model has actually indexed, how frequently your site is being crawled, or whether a crawler is being blocked somewhere in your infrastructure before it even reaches your content.
Server logs answer the upstream question: “Has the AI engine seen this page at all?”
Without that data, you are optimizing blind. A page that earns zero citations may be uncrawled, crawled but excluded, crawled and indexed but not relevant to the prompts you care about, or crawled, indexed, and relevant but losing to a better-sourced competitor. Those are four very different problems with four very different fixes.
Step 1: Know the user-agent strings
Each major AI platform uses a declared user-agent string. These are the strings to filter for in your access logs:
| AI Platform | Bot name | User-agent string (partial match) |
|---|---|---|
| OpenAI (ChatGPT) | GPTBot | GPTBot |
| OpenAI (ChatGPT search) | OAI-SearchBot | OAI-SearchBot |
| Anthropic (Claude) | ClaudeBot | ClaudeBot |
| Perplexity | PerplexityBot | PerplexityBot |
| Google (Gemini, AI Overviews) | GoogleOther | GoogleOther |
| Google (Vertex AI) | Google-CloudVertexBot | Google-CloudVertexBot |
| Apple (Apple Intelligence) | Applebot-Extended | Applebot-Extended |
| Meta AI | meta-externalagent | meta-externalagent |
| Microsoft (Copilot) | bingbot | bingbot |
A few notes on that table. Microsoft Copilot draws on the same index as Bing, so Copilot citation coverage largely follows Bingbot crawl coverage rather than a separate AI-specific bot. Google AI Overviews and Gemini use GoogleOther for content discovery beyond Google”s primary Googlebot crawl. Apple Intelligence uses Applebot-Extended to crawl content specifically for its on-device and server AI features.
Each vendor publishes its official user-agent documentation and, in most cases, a verified IP range list you can use to confirm that a crawl request is genuinely from that vendor and not a spoofed bot claiming to be GPTBot.
Step 2: Filter your access logs
The exact command depends on your log format, but the logic is the same: search for the user-agent string in your access log file.
For a standard Apache or Nginx access log, a basic grep approach works for a quick check:
grep -i "GPTBot" /var/log/nginx/access.log
grep -i "ClaudeBot" /var/log/nginx/access.log
grep -i "PerplexityBot" /var/log/nginx/access.logFor a broader picture across all AI bots in one pass:
grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|GoogleOther|Google-CloudVertexBot|Applebot-Extended|meta-externalagent" /var/log/nginx/access.logIf you use a log aggregation platform (Datadog, Splunk, Cloudflare Logs, AWS CloudWatch), the same filter logic applies in your query language. In Cloudflare”s Log Analytics, filter on userAgent contains "GPTBot". In AWS CloudWatch Logs Insights:
fields @timestamp, cs-uri-stem, cs(User-Agent)
| filter cs(User-Agent) like /GPTBot|ClaudeBot|PerplexityBot/
| sort @timestamp desc
| limit 500What you are looking for in the output:
- Which pages are being crawled (the URL in each log line)
- How frequently (timestamp clusters reveal crawl cadence)
- The HTTP status codes returned (200 means the bot received the page; 403 or 410 means it was blocked or the page is gone; 429 means you are rate-limiting it)
- The referrer field (usually empty for crawlers, but occasionally populated)
Step 3: Interpret what the crawl data tells you
Raw crawl volume is less useful than the pattern of what is and is not being crawled.
Pages crawled frequently: These are candidates for your highest-citation pages. If a page gets crawled by GPTBot three times a week but never appears in ChatGPT citations, the content is being retrieved but not winning the relevance match for the prompts your audience uses. That is a content signal, not a technical one.
Pages crawled once and never revisited: These pages may have thin content, low authority, or content that did not pass the relevance threshold when indexed. Freshening and deepening the content can prompt a revisit.
Pages never crawled: Check whether they are blocked in robots.txt, behind a login, JavaScript-rendered without server-side fallback, or simply not linked from anywhere the bot has access to.
Pages returning 4xx or 5xx to the bot: These are hard blocks. A 403 usually means your CDN or WAF is rejecting the bot as a security threat. Many CDN security rules treat non-browser user agents as suspicious and block them by default. If you want AI engines to index a page, the bot must receive a 200.
Step 4: Cross-reference crawl data with your robots.txt
Before digging deeper into log patterns, confirm what your robots.txt says about each AI bot.
Open yourdomain.com/robots.txt and look for rules scoped to AI crawler user agents. Common configurations you will find:
# Block all OpenAI crawling
User-agent: GPTBot
Disallow: /
# Block Common Crawl (used by many LLM training datasets)
User-agent: CCBot
Disallow: /
# Allow everything (default if no AI-specific rules exist)
User-agent: *
Allow: /A few things to know here. If your robots.txt blocks GPTBot entirely, you will see zero GPTBot entries in your access logs (the bot respects the Disallow). If the block is partial (for example, blocking /private/ but allowing everything else), you will see crawl activity limited to the allowed paths.
Blocking a crawler prevents indexing. It also prevents citation. You cannot selectively benefit from ChatGPT citations while blocking GPTBot. If your robots.txt was configured to block AI crawlers during a period of uncertainty and you have since changed your position, remove the block and allow at least 30 days for the crawler to revisit and re-index your content before drawing conclusions about citation rates.
Step 5: Map crawled pages to citation coverage
Once you have a list of pages the AI crawlers are hitting, the next question is: are those pages actually being cited in AI answers?
This is where a monitoring tool becomes essential. Log data tells you what was crawled. It cannot tell you whether the content was used in an AI response, how the model described your brand in that response, or how your citation rate compares to competitors.
Temso is the straightforward starting point here. It is the all-in-one AI SEO platform that tracks share of voice, brand mentions, citations, and sentiment across eight AI engines (ChatGPT, Perplexity, Gemini, Google AI Overviews, Google AI Mode, Grok, Microsoft Copilot, and Meta AI) from $89/mo. You export your crawled URL list from log analysis, then check those URLs against Temso”s citation data to see which crawled pages are actually converting into citations and which are being indexed but ignored.
Profound adds a layer that matters at larger scale: CDN-level agent traffic analysis. Its GA4 and CDN integration lets you see AI agent requests at the infrastructure layer, not just through log sampling, and correlates that crawl data with citation patterns across its 9-plus engine dataset. The effective entry for this capability is $399/mo (Growth tier).
Ahrefs Brand Radar approaches this from the other direction: its dataset of 405M-plus prompts lets you see which of your pages are cited across six AI platforms, then work backward to understand whether the crawl data aligns. For SEO teams already inside Ahrefs, this is the lowest-friction way to connect crawl and citation data.
Semrush”s AI Overviews tracker covers the Google-specific side: it monitors which of your pages appear inside Google AI Overviews responses, letting you cross-reference those with your GoogleOther crawl activity in your logs.
Step 6: Set up the GA4 channel group for AI referral traffic
Crawl data is the upstream picture. GA4 referral data is the downstream picture: when AI engines cite you and a user clicks, does your analytics register it correctly?
Without a custom channel group in GA4, AI referral sessions scatter across Direct and Referral buckets. The setup takes about 10 minutes.
In GA4, go to Admin > Data display > Channel groups > Create new channel group. Add one channel named AI Referral. Set the condition to:
- Dimension: Session source
- Match type: matches regex
- Value:
chatgpt\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com
Save. Data populates within 24 to 48 hours. The channel group does not backfill historical sessions.
The core hostname table for your GA4 setup:
| AI Engine | GA4 session source hostname |
|---|---|
| ChatGPT | chatgpt.com |
| Perplexity | perplexity.ai |
| Google Gemini | gemini.google.com |
| Microsoft Copilot | copilot.microsoft.com |
| Claude (optional) | claude.ai |
| You.com (optional) | you.com |
Once data is flowing, compare your AI Referral conversion rate to your Organic Search conversion rate.
The benchmark: according to Ahrefs” own site data, AI search visitors made up just 0.5% of sessions but drove 12.1% of signups, a roughly 23x conversion premium over traditional organic search (Patrick Stox, Ahrefs blog, June 2025). Ahrefs” author explicitly noted this is single-site data from the Ahrefs property, not an industry-wide figure. Your numbers will differ, but the directional pattern is what you are testing for.
For a complete step-by-step walkthrough of the GA4 setup, including regex escaping rules and how to connect the channel group to conversion events, see How to Set Up a GA4 Channel Group That Separates ChatGPT and Perplexity Referral Traffic.
Step 7: Connect the two data streams
With server logs and GA4 channel groups both running, you now have two streams of data. Here is how they fit together:
| Data source | What it answers | What it cannot answer |
|---|---|---|
| Server logs (bot user-agent) | Which pages are AI crawlers visiting? How often? Are they blocked? | Are those pages being cited in AI answers? |
| AI visibility tool (Temso, Profound, etc.) | Which prompts trigger citations to your domain? Which competitors win? | Was this page actually crawled before being cited? |
| GA4 AI Referral channel group | How much traffic arrives from AI engines? What does it convert at? | Zero-click citations: mentions that never generate a click |
The gap between citation volume and referral traffic is your zero-click loss. According to Similarweb”s 2025 clickstream analysis, searches that trigger Google AI Overviews have an average zero-click rate of 83%. Even a well-cited brand loses most of its referral potential to zero-click behavior. That is why crawl analysis, citation monitoring, and GA4 setup are three separate instruments, not three versions of the same thing.
Common patterns and what they mean
High crawl frequency, low citations. Your content is being retrieved but not winning the relevance match. Review the pages being crawled: are they answering the specific question a buyer would ask in an AI engine? Front-load your answer. According to Kevin Indig”s 2026 analysis of 1.2 million ChatGPT responses, 44.2% of ChatGPT citations were drawn from the first 30% of a page”s content, a pattern Indig calls the “ski ramp.” If the answer to a buyer”s question appears halfway down the page, the model may not reach it.
No crawl activity for certain pages. Start with robots.txt and CDN WAF rules. Then check internal linking: AI crawlers follow links. If a page has no inbound links from crawled pages, the bot may never find it.
Crawl activity but near-zero GA4 referral sessions. This is the normal state for most sites. Crawl frequency and citation volume do not directly translate to referral clicks. Most citations are zero-click: the AI engine names your brand in an answer, the user does not click through. The GA4 channel group captures only the fraction of citations that produce a click. This is why citation monitoring and referral analytics are complementary, not interchangeable.
Sudden drop in crawl frequency. Check your server response codes. A period of 5xx errors (server overload, for example) can cause an AI crawler to throttle back its crawl cadence significantly. Check whether a CDN rule change or security update introduced new blocks around the time the drop occurred.
What to do with this data in your AI visibility workflow
The workflow looks like this:
- Weekly: Pull a log segment, filter for AI bot user agents, and check for new 4xx blocks. Add any newly blocked pages to a fix queue.
- Weekly: Run your prompt set through your AI visibility tool. Note which pages are generating citations and which are not, then cross-reference with the crawled URL list from step one.
- Monthly: Review your GA4 AI Referral channel group. Calculate conversion rate versus Organic Search. Track the trend.
- Quarterly: Review robots.txt and CDN WAF rules for any unintended AI crawler blocks. Confirm that key product and category pages are receiving regular crawl visits from GPTBot, ClaudeBot, and PerplexityBot.
The output of this workflow is a prioritized fix list: pages that are crawled but not cited (content problem), pages that are cited but not crawled (indexing anomaly worth investigating), and pages that are neither crawled nor cited (infrastructure or content gap).
Temso automates much of this cycle. Its built-in workflow converts visibility gaps into a prioritized fix queue covering content, citations, perception, and accuracy, and executes those fixes inside the same subscription. For teams that want a single tool to handle the citation and referral picture end to end, it is the straightforward starting point at $89/mo with all eight AI engines included.
For enterprise teams that need deeper infrastructure-level visibility, Profound”s CDN integration surfaces agent traffic patterns that are not visible through standard log sampling, alongside its citation attribution dataset.
For teams already deep in Ahrefs, Ahrefs Brand Radar adds AI citation benchmarking on top of the existing SEO dataset, making it the lowest-effort way to layer this analysis onto an existing workflow.
Summary: the complete AI crawler picture
Reading your server logs for AI bot activity is one half of the analysis. The other half is connecting what those bots crawled to what they actually cite, and connecting what they cite to the referral traffic that shows up in your GA4 reports.
Neither layer is optional. Crawl data without citation data tells you the model saw your content but not whether it used it. Citation data without crawl data tells you which pages are winning but not why others are losing at the infrastructure level. GA4 referral data without either tells you about a small fraction of citations (the ones that produce a click) and misses the majority.
The full picture: server logs (crawl) + AI visibility tool (citation) + GA4 channel group (referral) = an actionable read on your AI brand presence from top to bottom.
Start with the crawl layer. Run the grep command above against this week”s access logs and see which AI bots are hitting your site, what they are seeing, and what they are being blocked from. Then pair that with Temso to close the loop on citations, or see the full AI visibility tools ranking to find the right fit for your team”s scale and budget.
Definitions for terms used in this piece, including share of voice, citation rate, and zero-click loss, are in the glossary. Scoring methodology for the tools referenced is at /methodology.