Skip to content
Technical files

AI crawler log analyzer

This AI crawler log analyzer reads your server access log and reports which AI bots fetched your pages, how often, and which pages they took. It also finds people who arrived from an AI answer. Your browser reads the file and never uploads it, so nothing leaves your machine.

By The Promptscope team (Product and testing). Last updated .

Log

Drop your access log here

Or choose a file. Nothing is uploaded: the file is read in this browser tab and never sent anywhere. You can check that in your network tab.

AI crawler log analyzer diagram: the log stays on your computer and the browser reads it
How this page works. There is no upload step, because there is no endpoint to upload to.
Steps

How to analyze your log for AI crawlers, step by step

  1. Download an access log from your server, usually access.log on Nginx or Apache, or an export from Cloudflare.
  2. Then drop the file onto the box above, or choose it from disk.
  3. Wait while your browser reads it. We receive nothing, so even a large file stays on your machine.
  4. Finally, read which bots came, which pages they took, and whether any request failed.

The first 3 runs each day are free. After that, see the pricing page for packs and plans, or read how we test the tool.

Why an AI crawler log analyzer beats any estimate

Every other measure in this market is a sample. A prompt test asks a model some questions and counts the answers, which tells you about those questions and nothing else. Your log is different, because it records what actually happened on your own server.

So the log answers a narrower question, but answers it exactly. GPTBot either fetched your pricing page on Tuesday or it did not. There is no inference, no estimate and no sampling error in that.

What the AI crawler log analyzer looks for

First, the named crawlers: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, ClaudeBot and Claude-SearchBot for Anthropic, PerplexityBot, Google-Extended, Applebot-Extended, and several more including Bytespider, Amazonbot and CCBot.

Second, referrers. When a person clicks a link inside an AI answer, their visit usually carries a referrer such as chatgpt.com or perplexity.ai. We count those separately, because a crawler fetching a page and a human reading it are different events. The agents ending in User matter most here, since they only appear when someone followed a link.

Nothing is uploaded, and you can check that

An access log holds full IP addresses, referrers and sometimes tokens sitting in query strings. Sending one to a third party is a real privacy decision, and in most companies the security team would refuse it.

Therefore this tool does not ask for it. Your browser reads the file, counts what it finds and draws the report. Open your network tab while it runs and you will see no request go out. We also drop query strings before counting, so a token in a URL never reaches even the report on your screen.

What an AI crawler log analyzer cannot tell you

An AI crawler log analyzer cannot tell you which questions people asked. No engine publishes them, and a log records the visit rather than the conversation behind it. Anyone who sells you real user prompts is inferring them.

It also cannot prove a sale. A referrer shows a visit, not a decision. Treat these numbers as a leading indicator, and read them next to the readiness report, because a crawler that never appears in your log may be blocked rather than uninterested.

What each line of a log can tell you

What each line of a log can tell you
SignalWhat it means
GPTBot, ClaudeBot, PerplexityBotthe crawler took the page for its index or training
ChatGPT-User, Claude-User, Perplexity-Usersomeone followed a link from a conversation
Referrer chatgpt.com or perplexity.aia real visit that came out of an AI answer
A 4xx answer to a crawlerit asked and did not get the page, which is worth fixing
A bot that never appearssomething blocks it, or it has not found the site yet
Pages taken most oftenwhat the engines treat as the useful part of your site
Questions

AI crawler log analyzer FAQ

Is my log file uploaded anywhere?

No. There is no upload endpoint behind this page. Your browser opens the file, counts what is in it and renders the report locally. You can confirm it by opening your network tab before you drop the file in.

Which log formats does the AI crawler log analyzer read?

The combined format that Apache and Nginx write by default, and JSON lines as Cloudflare and several hosts emit. It counts the lines it cannot read and tells you the number, rather than dropping them quietly.

How big a file can it handle?

It reads the file in slices rather than loading it whole, so a few hundred megabytes is fine on a normal laptop. It stays responsive while it works, and a bigger file simply takes longer.

No AI crawler appears in my log. What does that mean?

Usually one of two things. Either your robots.txt or a firewall rule turns them away before your server records anything, or the site is new enough that they have not come yet. Run the readiness check first, because a blocked crawler explains an empty log immediately.

Can I see which prompts led people to my site?

No, and no tool can. The engines do not publish the questions, and your server only records the visit. What you can see is which pages get fetched and which get clicked through, which is the useful half of the question.

Next
References

Sources and references