AI crawlers are not always visible in Google Analytics, but they leave a trace in the server logs. That is exactly where you can check whether your site is visited by the bots of OpenAI, Anthropic, Perplexity, and other AI systems, which URLs they fetch, which IPs they use, and whether their activity does not burden the infrastructure. For GEO and technical SEO efforts, log analysis is becoming one of the simplest ways to assess whether content is really accessible to AI assistants.

The biggest mistake today is treating all AI bots the same way. Some of them serve to train models, some to search for and cite content in answers, and some work only when the user pastes a specific URL into the chat. Blocking one type of bot can protect content from training use, while blocking another can limit the site’s visibility in AI answers.

Why Are Server Logs More Important Than GA4 When Analyzing AI Bots?

Google Analytics, Matomo, or other classic analytics tools measure primarily the behavior of users who run the tracking code in the browser. AI crawlers work differently: they fetch a resource from the server, often without loading JavaScript, without a user session, and without a standard analytics event. As a result, they can intensively visit the site but remain invisible in the traffic reports.

Server logs are a technical record of every HTTP request. Depending on the server configuration, they contain, among other things, the date, IP, request method, fetched URL, response code, response size, referrer, and the User-Agent header. The last of these allows you to preliminarily recognize a bot, e.g. GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot.

From the GEO perspective, logs answer three practical questions:

  • whether AI systems fetch content from the domain at all;
  • which sections of the site are visited most often;
  • whether the server, CDN, or WAF is not blocking bots that theoretically have access in robots.txt.

The last of these is particularly important. A site can have a correct robots.txt, and at the same time Cloudflare, Akamai, firewall rules, or request limiting can return 403 or 429 errors to bots. From the perspective of visibility in AI, the effect is similar: the crawler does not fetch the content, so the model has a smaller chance of citing it.

Which Types of AI Crawlers Are Worth Distinguishing?

In technical SEO, the most important thing is separating bots by their purpose of operation, and not just by their owner. OpenAI, Anthropic, and Perplexity use different User-Agents for different tasks. That is why the same domain can simultaneously block a training bot and allow a search bot.

OpenAI

User-Agent / botFunctionEffect of blocking
GPTBotfetching content that may support model traininglimiting the use of content for training OpenAI models
OAI-SearchBotsearching and indexing for ChatGPT Search featuresa smaller chance of the site appearing in ChatGPT Search answers
ChatGPT-Userfetching the page at the user’s request, e.g. after pasting a URLmaking it harder for users to analyze a specific URL in ChatGPT

Anthropic

User-Agent / botFunctionEffect of blocking
ClaudeBotfetching content that may support the development of Claude modelslimiting the content’s share in Anthropic’s future training sets
Claude-SearchBotfetching content for the quality and relevance of Claude search resultslower visibility in Claude results and answers based on web search
Claude-Userfetching content at the request of a Claude userlimiting the ability to fetch the page in response to a user’s query

Perplexity

User-Agent / botFunctionEffect of blocking
PerplexityBotfetching content for indexing and Perplexity answersa smaller chance of citation in Perplexity
Perplexity-Userfetching content linked to a user’s actioncontrol also requires a server/CDN/WAF layer

Such a distinction shows why the general command “block AI bots” is too broad. If a company wants to protect content from model training while at the same time keeping a chance of citation in AI answers, a better solution is a separate policy for training and search bots.

How Do You Find AI Crawlers in the Server Logs?

The simplest analysis starts with fetching the raw logs. In shared hosting, they may be available in the panel as “Raw Access Logs”. On servers with SSH access, typical locations are:

/var/log/nginx/access.log
 /var/log/apache2/access.log

As a first step, it is enough to filter the logs by the most important User-Agents:

grep “GPTBot” /var/log/nginx/access.log
 grep “OAI-SearchBot” /var/log/nginx/access.log
 grep “ClaudeBot” /var/log/nginx/access.log
 grep “Claude-SearchBot” /var/log/nginx/access.log
 grep “PerplexityBot” /var/log/nginx/access.log

For a quick review of several bots at once, you can use:

egrep “GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User” /var/log/nginx/access.log

An example log entry might look like this:

192.0.2.15 – – [04/Oct/2026:14:32:10 +0200] “GET /blog/geo-a-seo/ HTTP/1.1” 200 18492 “-” “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot”

The User-Agent alone, however, is not enough for a final verification. Any script can impersonate GPTBot or ClaudeBot, which is why, after the initial filtering, you should check the IP.

How Do You Verify Whether a Bot Is Real?

Verifying a bot’s authenticity protects against spoofing, that is, falsifying the User-Agent header. The simplest variant is Reverse DNS:

host 192.0.2.15

In the case of OpenAI, the result should point to a domain belonging to OpenAI, and then the address can additionally be compared with the official IP lists published by the operator. Analogously, in the case of Anthropic, you should use the current documentation and source lists, and with Perplexity — the official ranges and observations from the CDN/WAF layer.

A good audit minimum looks as follows:

  1. Extract all requests with AI User-Agents.
  2. Group them by IP, User-Agent, response code, and URL.
  3. Verify the IP for the most active bots.
  4. Mark the requests as a real bot, a suspicious bot, or an unrecognized automaton.
  5. Check whether the bots receive 200 codes or whether they fall into 403, 404, 429, 500, 502, or 504.

Only such a data set tells you whether the site is really accessible to AI crawlers. The entry in robots.txt alone does not show whether the bot can actually fetch the HTML.

How Do You Interpret the Activity of AI Crawlers?

The first reflex often comes down to the question: “was the bot on the site?”. A better question is: “what exactly did it fetch and with what result?”. The mere presence of GPTBot or ClaudeBot does not yet mean that the site will be visible in AI answers.

In the logs, you should assess several layers:

Analysis areaWhat to checkWhat a problem means
Accessibility200 codes for important URLsthe bot is not blocked and can fetch the content
Blocks403, 429, 503 codesa WAF, rate limit, or the server restricts access
Crawl scopedirectories and types of subpagesthe bot may skip service pages, the blog, or technical resources
Frequencynumber of requests per minute/hoursudden spikes can burden the server
Updatesreturns to new and changed URLsthe bot discovers new content or uses the sitemap
Crawl/referral relationbot fetches vs entries from AIhigh crawl with no traffic may require limits

It is also important to distinguish intent. Heavy GPTBot traffic may mean fetching content for training purposes, but it does not have to translate into entries from ChatGPT. In turn, the activity of OAI-SearchBot is more closely associated with search and visibility in ChatGPT Search answers.

How Do You Set Up robots.txt Without Losing Visibility in AI?

The safest model is separate rules for training, search, and user-triggered bots. An example scheme for a company that wants to keep visibility in AI answers but limit the content’s share in model training might look like this:

# OpenAI – visibility in ChatGPT Search
 User-agent: OAI-SearchBot
 Allow: /
 
 # OpenAI – limiting model training
 User-agent: GPTBot
 Disallow: /
 
 # Anthropic – visibility in Claude search
 User-agent: Claude-SearchBot
 Allow: /
 
 # Anthropic – limiting model training
 User-agent: ClaudeBot
 Disallow: /
 
 # Perplexity – visibility in Perplexity answers
 User-agent: PerplexityBot
 Allow: /

This is not a universal pattern for every industry. News services, knowledge bases, portals with paid access, and stores with unique product data may adopt a more restrictive policy. Service, B2B, and e-commerce companies that care about being cited by AI usually should not block all bots with a single User-agent: * Disallow: / entry.

The configuration should be tested in practice. After changing robots.txt, check whether the bots still fetch the most important URLs and whether errors do not appear on the CDN or WAF side. In many cases, the problem is not the robots.txt file itself, but additional protections that block bots despite the declared Allow.

Which Mistakes Most Often Distort Log Analysis?

The most common mistake is equating every User-Agent with a real bot. Without IP verification, you can confuse an official crawler with a cheap scraper impersonating a well-known name.

The second mistake is analyzing too short a period. A one-day slice of logs may not show the real pattern, because AI crawlers often work in waves. For a meaningful analysis, a minimum of 30 days is useful, and in larger services it is better to compare months.

The third mistake is looking solely at the number of requests. If a bot fetched 500 URLs but most of them returned 404 or 429, the conclusion is completely different than with 500 correct 200 responses. For GEO, the quality of access to the most important content is more important than the scale of crawling itself.

The fourth mistake is blocking bots without distinguishing their purpose. Blocking a training bot and a search bot has different effects. For visibility in AI, the most dangerous thing is accidentally cutting off those crawlers that are responsible for current search, citations, and fetching pages at the user’s request.

How Do You Measure the Effectiveness of Log Analysis for GEO?

Log analysis should end not only with a list of bots, but with a set of indicators to monitor. Thanks to this, you can assess whether changes in robots.txt, the WAF, and the site structure improve content accessibility for AI.

MetricWhat it measuresHow to measureTarget interpretation
AI crawler coveragethe share of important URLs visited by AI botscomparison of the list of strategic URLs with the bot logsthe most important service and guide pages are fetched
AI bot success ratethe percentage of bot requests ending with a 200 codeserver logs, filter by User-Agent and response codea low number of 403/429/5xx for bots that are supposed to have access
Crawl-to-refer ratiothe relation of bot fetches to entries from AI assistantslogs + referral data from analytics or logshigh crawl with no traffic requires controlling cost and load
AI crawl wastefetches of low-value URLsanalysis of directories, parameters, tags, paginationbots do not use resources on junk addresses
Bot load impactthe impact of bots on response time and server errorscorrelation of bot visits with TTFB, 429, 502, 504crawlers do not worsen the site’s operation for users

In practice, it is best to prepare a simple spreadsheet: bot, bot type, number of requests, number of unique URLs, number of errors, most frequently fetched directories, most frequent IPs, verification status, and recommended action.

What Should You Do After the First Audit?

After the first review of the logs, three types of actions usually appear. The first concerns accessibility: fixing 4xx/5xx errors, making important content available in HTML, updating the sitemap, and making sure that crawlers do not land only on random addresses.

The second concerns control: separating the bots in robots.txt, setting frequency limits for aggressive crawlers, and verifying the WAF rules. Not every AI activity should be blocked, but every one is worth measuring.

The third concerns visibility: comparing the logs with the results of tests in ChatGPT, Claude, and Perplexity. If the bot fetches the page but AI does not cite the brand, the problem may lie not in crawlability, but in the content structure, the source’s authority, or the lack of external confirmations.

How does GPTBot differ from OAI-SearchBot?

GPTBot is an OpenAI bot associated with fetching content that may be used for developing models. OAI-SearchBot is responsible for searching and the visibility of pages in ChatGPT’s search functions. You can block GPTBot while allowing access to OAI-SearchBot.

Why does GA4 not show visits from AI bots?

AI crawlers usually fetch raw HTML and do not run JavaScript scripts. GA4 is based on tracking code, so such a visit does not create a standard analytics session. That is why server logs are needed to detect bots.

How do you check whether Claude visits the site?

In the server logs, look for, among others, ClaudeBot, Claude-SearchBot, and Claude-User. After finding the entries, verify the IP and check what response codes the bot receives.

Is blocking AI bots in robots.txt always good?

No. Blocking can protect content from training use, but it can also limit visibility in AI answers. It is best to make the decision separately for training, search, and user-triggered bots.

What does a high crawl-to-refer ratio mean?

A high ratio means that the bot fetches many pages relative to the number of users referred from a given AI system. This may be acceptable for visibility efforts, but with a heavy server load it requires rate limiting, cache, or a change to the WAF rules.

Sources