AI Crawlers in Server Logs – How Do You Check Whether ChatGPT, Claude, and Perplexity Visit Your Site?
AI crawlers are not always visible in Google Analytics, but they leave a trace in the server logs. That is exactly where you can check whether your site is visited by the bots of OpenAI, Anthropic, Perplexity, and other AI systems, which URLs they fetch, which IPs they use, and whether their activity does not burden the infrastructure. For GEO and technical SEO efforts, log analysis is becoming one of the simplest ways to assess whether content is really accessible to AI assistants.
The biggest mistake today is treating all AI bots the same way. Some of them serve to train models, some to search for and cite content in answers, and some work only when the user pastes a specific URL into the chat. Blocking one type of bot can protect content from training use, while blocking another can limit the site’s visibility in AI answers.
Why Are Server Logs More Important Than GA4 When Analyzing AI Bots?
Google Analytics, Matomo, or other classic analytics tools measure primarily the behavior of users who run the tracking code in the browser. AI crawlers work differently: they fetch a resource from the server, often without loading JavaScript, without a user session, and without a standard analytics event. As a result, they can intensively visit the site but remain invisible in the traffic reports.
Server logs are a technical record of every HTTP request. Depending on the server configuration, they contain, among other things, the date, IP, request method, fetched URL, response code, response size, referrer, and the User-Agent header. The last of these allows you to preliminarily recognize a bot, e.g. GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot.
From the GEO perspective, logs answer three practical questions:
- whether AI systems fetch content from the domain at all;
- which sections of the site are visited most often;
- whether the server, CDN, or WAF is not blocking bots that theoretically have access in robots.txt.
The last of these is particularly important. A site can have a correct robots.txt, and at the same time Cloudflare, Akamai, firewall rules, or request limiting can return 403 or 429 errors to bots. From the perspective of visibility in AI, the effect is similar: the crawler does not fetch the content, so the model has a smaller chance of citing it.
Which Types of AI Crawlers Are Worth Distinguishing?
In technical SEO, the most important thing is separating bots by their purpose of operation, and not just by their owner. OpenAI, Anthropic, and Perplexity use different User-Agents for different tasks. That is why the same domain can simultaneously block a training bot and allow a search bot.
OpenAI
| User-Agent / bot | Function | Effect of blocking |
| GPTBot | fetching content that may support model training | limiting the use of content for training OpenAI models |
| OAI-SearchBot | searching and indexing for ChatGPT Search features | a smaller chance of the site appearing in ChatGPT Search answers |
| ChatGPT-User | fetching the page at the user’s request, e.g. after pasting a URL | making it harder for users to analyze a specific URL in ChatGPT |
Anthropic
| User-Agent / bot | Function | Effect of blocking |
| ClaudeBot | fetching content that may support the development of Claude models | limiting the content’s share in Anthropic’s future training sets |
| Claude-SearchBot | fetching content for the quality and relevance of Claude search results | lower visibility in Claude results and answers based on web search |
| Claude-User | fetching content at the request of a Claude user | limiting the ability to fetch the page in response to a user’s query |
Perplexity
| User-Agent / bot | Function | Effect of blocking |
| PerplexityBot | fetching content for indexing and Perplexity answers | a smaller chance of citation in Perplexity |
| Perplexity-User | fetching content linked to a user’s action | control also requires a server/CDN/WAF layer |
Such a distinction shows why the general command “block AI bots” is too broad. If a company wants to protect content from model training while at the same time keeping a chance of citation in AI answers, a better solution is a separate policy for training and search bots.
How Do You Find AI Crawlers in the Server Logs?
The simplest analysis starts with fetching the raw logs. In shared hosting, they may be available in the panel as “Raw Access Logs”. On servers with SSH access, typical locations are:
/var/log/nginx/access.log
/var/log/apache2/access.log
As a first step, it is enough to filter the logs by the most important User-Agents:
grep “GPTBot” /var/log/nginx/access.log
grep “OAI-SearchBot” /var/log/nginx/access.log
grep “ClaudeBot” /var/log/nginx/access.log
grep “Claude-SearchBot” /var/log/nginx/access.log
grep “PerplexityBot” /var/log/nginx/access.log
For a quick review of several bots at once, you can use:
egrep “GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User” /var/log/nginx/access.log
An example log entry might look like this:
192.0.2.15 – – [04/Oct/2026:14:32:10 +0200] “GET /blog/geo-a-seo/ HTTP/1.1” 200 18492 “-” “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot”
The User-Agent alone, however, is not enough for a final verification. Any script can impersonate GPTBot or ClaudeBot, which is why, after the initial filtering, you should check the IP.
How Do You Verify Whether a Bot Is Real?
Verifying a bot’s authenticity protects against spoofing, that is, falsifying the User-Agent header. The simplest variant is Reverse DNS:
host 192.0.2.15
In the case of OpenAI, the result should point to a domain belonging to OpenAI, and then the address can additionally be compared with the official IP lists published by the operator. Analogously, in the case of Anthropic, you should use the current documentation and source lists, and with Perplexity — the official ranges and observations from the CDN/WAF layer.
A good audit minimum looks as follows:
- Extract all requests with AI User-Agents.
- Group them by IP, User-Agent, response code, and URL.
- Verify the IP for the most active bots.
- Mark the requests as a real bot, a suspicious bot, or an unrecognized automaton.
- Check whether the bots receive 200 codes or whether they fall into 403, 404, 429, 500, 502, or 504.
Only such a data set tells you whether the site is really accessible to AI crawlers. The entry in robots.txt alone does not show whether the bot can actually fetch the HTML.
How Do You Interpret the Activity of AI Crawlers?
The first reflex often comes down to the question: “was the bot on the site?”. A better question is: “what exactly did it fetch and with what result?”. The mere presence of GPTBot or ClaudeBot does not yet mean that the site will be visible in AI answers.
In the logs, you should assess several layers:
| Analysis area | What to check | What a problem means |
| Accessibility | 200 codes for important URLs | the bot is not blocked and can fetch the content |
| Blocks | 403, 429, 503 codes | a WAF, rate limit, or the server restricts access |
| Crawl scope | directories and types of subpages | the bot may skip service pages, the blog, or technical resources |
| Frequency | number of requests per minute/hour | sudden spikes can burden the server |
| Updates | returns to new and changed URLs | the bot discovers new content or uses the sitemap |
| Crawl/referral relation | bot fetches vs entries from AI | high crawl with no traffic may require limits |
It is also important to distinguish intent. Heavy GPTBot traffic may mean fetching content for training purposes, but it does not have to translate into entries from ChatGPT. In turn, the activity of OAI-SearchBot is more closely associated with search and visibility in ChatGPT Search answers.
How Do You Set Up robots.txt Without Losing Visibility in AI?
The safest model is separate rules for training, search, and user-triggered bots. An example scheme for a company that wants to keep visibility in AI answers but limit the content’s share in model training might look like this:
# OpenAI – visibility in ChatGPT Search
User-agent: OAI-SearchBot
Allow: /
# OpenAI – limiting model training
User-agent: GPTBot
Disallow: /
# Anthropic – visibility in Claude search
User-agent: Claude-SearchBot
Allow: /
# Anthropic – limiting model training
User-agent: ClaudeBot
Disallow: /
# Perplexity – visibility in Perplexity answers
User-agent: PerplexityBot
Allow: /
This is not a universal pattern for every industry. News services, knowledge bases, portals with paid access, and stores with unique product data may adopt a more restrictive policy. Service, B2B, and e-commerce companies that care about being cited by AI usually should not block all bots with a single User-agent: * Disallow: / entry.
The configuration should be tested in practice. After changing robots.txt, check whether the bots still fetch the most important URLs and whether errors do not appear on the CDN or WAF side. In many cases, the problem is not the robots.txt file itself, but additional protections that block bots despite the declared Allow.
Which Mistakes Most Often Distort Log Analysis?
The most common mistake is equating every User-Agent with a real bot. Without IP verification, you can confuse an official crawler with a cheap scraper impersonating a well-known name.
The second mistake is analyzing too short a period. A one-day slice of logs may not show the real pattern, because AI crawlers often work in waves. For a meaningful analysis, a minimum of 30 days is useful, and in larger services it is better to compare months.
The third mistake is looking solely at the number of requests. If a bot fetched 500 URLs but most of them returned 404 or 429, the conclusion is completely different than with 500 correct 200 responses. For GEO, the quality of access to the most important content is more important than the scale of crawling itself.
The fourth mistake is blocking bots without distinguishing their purpose. Blocking a training bot and a search bot has different effects. For visibility in AI, the most dangerous thing is accidentally cutting off those crawlers that are responsible for current search, citations, and fetching pages at the user’s request.
How Do You Measure the Effectiveness of Log Analysis for GEO?
Log analysis should end not only with a list of bots, but with a set of indicators to monitor. Thanks to this, you can assess whether changes in robots.txt, the WAF, and the site structure improve content accessibility for AI.
| Metric | What it measures | How to measure | Target interpretation |
| AI crawler coverage | the share of important URLs visited by AI bots | comparison of the list of strategic URLs with the bot logs | the most important service and guide pages are fetched |
| AI bot success rate | the percentage of bot requests ending with a 200 code | server logs, filter by User-Agent and response code | a low number of 403/429/5xx for bots that are supposed to have access |
| Crawl-to-refer ratio | the relation of bot fetches to entries from AI assistants | logs + referral data from analytics or logs | high crawl with no traffic requires controlling cost and load |
| AI crawl waste | fetches of low-value URLs | analysis of directories, parameters, tags, pagination | bots do not use resources on junk addresses |
| Bot load impact | the impact of bots on response time and server errors | correlation of bot visits with TTFB, 429, 502, 504 | crawlers do not worsen the site’s operation for users |
In practice, it is best to prepare a simple spreadsheet: bot, bot type, number of requests, number of unique URLs, number of errors, most frequently fetched directories, most frequent IPs, verification status, and recommended action.
What Should You Do After the First Audit?
After the first review of the logs, three types of actions usually appear. The first concerns accessibility: fixing 4xx/5xx errors, making important content available in HTML, updating the sitemap, and making sure that crawlers do not land only on random addresses.
The second concerns control: separating the bots in robots.txt, setting frequency limits for aggressive crawlers, and verifying the WAF rules. Not every AI activity should be blocked, but every one is worth measuring.
The third concerns visibility: comparing the logs with the results of tests in ChatGPT, Claude, and Perplexity. If the bot fetches the page but AI does not cite the brand, the problem may lie not in crawlability, but in the content structure, the source’s authority, or the lack of external confirmations.
GPTBot is an OpenAI bot associated with fetching content that may be used for developing models. OAI-SearchBot is responsible for searching and the visibility of pages in ChatGPT’s search functions. You can block GPTBot while allowing access to OAI-SearchBot.
AI crawlers usually fetch raw HTML and do not run JavaScript scripts. GA4 is based on tracking code, so such a visit does not create a standard analytics session. That is why server logs are needed to detect bots.
In the server logs, look for, among others, ClaudeBot, Claude-SearchBot, and Claude-User. After finding the entries, verify the IP and check what response codes the bot receives.
No. Blocking can protect content from training use, but it can also limit visibility in AI answers. It is best to make the decision separately for training, search, and user-triggered bots.
A high ratio means that the bot fetches many pages relative to the number of users referred from a given AI system. This may be acceptable for visibility efforts, but with a heavy server load it requires rate limiting, cache, or a change to the WAF rules.
Sources
- OpenAI Developers, “Overview of OpenAI Crawlers”.
- Anthropic Help Center, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”.
- Cloudflare Blog, “Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives”.
- Cloudflare, Verified Bots documentation and bot management materials.
- Documentation and technical materials on robots.txt.