What Is ClaudeBot and Should You Block It?

You found ClaudeBot in your server logs and now you want to know whether to let it keep reading. We have 100 days of our own logs on this exact question, so this post answers it with counts rather than opinions.

What is ClaudeBot?

ClaudeBot is Anthropic's training crawler. It makes ordinary HTTP GET requests to public URLs, reads the response, and uses the content to train Claude models. It cannot link back to you and it does not produce citations. Anthropic runs two other bots, Claude-User and Claude-SearchBot, that handle live retrieval and search indexing.

We log every AI bot request to cite.solutions at the edge. Between June 3 and September 10, 2026, ClaudeBot made 106,107 requests across 66,986 unique URLs. Over the same window, Claude-User made 172 and Claude-SearchBot made one.

That ratio is the whole decision. Everything below is the detail behind it, and it extends the cross-vendor picture we published in which AI crawlers get you cited.

What 106,107 ClaudeBot requests actually look like

Vendor pages describe ClaudeBot in the abstract. Here is what it did to one real site over 94 active crawl days, at an average of 1,129 requests per day.

Finding 1: ClaudeBot read 66,986 URLs and almost never came back

ClaudeBot averaged 1.58 requests per unique URL. It finds a page, takes it, and moves on. Compare that to OAI-SearchBot at 4.08 requests per URL, a bot whose job is to keep an index current.

PromptWatch measured the same pattern independently. In a 71,060-request analysis of a European real-estate site between August 3 and September 1, 2026, ClaudeBot issued 15,182 requests across 10,414 distinct property URLs, about 1.5 per page. Different site, different month, same behaviour.

ClaudeBot takes the page once and leaves. Plan for one read, not a relationship.

This matters for anyone who assumes a training crawler will notice a rewrite. It will not, at least not on any timeline you can plan around.

Finding 2: 97.6% of the crawl went to one section, and it was not the blog

We publish 317 blog posts. ClaudeBot spent 1.0% of its requests on them.

Where it actually went was the CITE Index, our dated, structured, per-prompt citation study: 96,607 requests, or 91% of everything. Add the study's brand images and it reaches 97.6%. Our homepage got four requests in 100 days. Our service pages got two or three each.

Section	ClaudeBot requests	Share	Unique URLs
CITE Index study pages	96,607	91.0%	65,388
Study brand images	6,925	6.5%	97
Blog posts	1,043	1.0%	636
Tag pages	1,023	1.0%	626
Service and other pages	374	0.4%	236
Homepage	4	0.004%	1

The study section is not bigger because it is better written. It is bigger because it generates thousands of dated, individually addressable, data-dense URLs. A training crawler optimising for volume of novel text will find that shape first.

If your site is a 40-page marketing site, ClaudeBot will be done with you in an afternoon.

Finding 3: ClaudeBot fetched a Markdown twin for every HTML page it took

Every page on our site has a Markdown version at the same URL with .md appended, advertised with a <link rel="alternate"> tag.

ClaudeBot used it. Of its 106,107 requests, 33,472 went to .md URLs, and in July, its heaviest crawl month, the share hit 45.9%. On the blog the pairing is exact: 317 unique HTML post URLs and 317 unique .md post URLs.

This sits next to a finding we published earlier that looks like its opposite. Across 500 million AI bot visits, only 408 requests ever fetched /llms.txt. Both are true, and the difference is discovery. A single file at the root that nothing links to gets ignored. A per-page alternate that every page declares in its head gets taken.

Crawlers do not read your directory. They follow what your pages declare.

Finding 4: Anthropic's answer-time bot is 0.16% of its crawl volume

Claude-User is the bot that fetches a page when a person asks Claude a question that needs the live web. It made 172 requests to 84 unique URLs across 69 separate days.

It also behaves nothing like ClaudeBot. Where ClaudeBot spent 1% of its attention on the blog, Claude-User spent 60%. It returned repeatedly to a small set of pages: our llms.txt analysis 19 times, our FAQ schema post 11 times, our statistics page 10 times.

One honest caveat that most write-ups skip. Of those 172 fetches, 126 carried a claude-code/ token in the user agent, meaning a developer agent session rather than the consumer Claude app. The consumer-app retrieval number is closer to 46 requests in 100 days.

The bot that can cite you is the one that barely shows up in your logs.

Finding 5: The only Claude-SearchBot hit we logged was a fake

We recorded exactly one request claiming to be Claude-SearchBot. It asked for /.env.

That is a credential-scanning probe wearing a crawler's name. The request came from 34.32.158.146, a Netherlands address that appears in none of the 26 prefixes Anthropic publishes at claude.com/crawling/bots.json. The string in a user agent header is free for anyone to type, which is exactly why that file exists.

A user agent is a claim, not an identity. Verify by IP before you count anything.

If you are building bot reporting on log strings alone, your ClaudeBot number is an upper bound, not a measurement. We cover the full method in our guide to running an AI crawler log audit.

Anthropic runs three bots and only one of them can cite you

Anthropic formalised this split in February 2026, and most robots.txt files we audit still treat all three as one thing. They do three different jobs and blocking them carries three different costs.

ClaudeBot collects training content and never links back

ClaudeBot feeds the next model. Its output surfaces months later as unattributed knowledge inside Claude's answers. Your brand may benefit from that, but nothing in your analytics will ever tell you so.

Claude-User fetches the page during a live conversation

Claude-User is the request that happens when someone asks Claude about your category and Claude goes to read. This is the fetch that produces a visible citation with your link in it. Anthropic's own documentation says blocking it "may reduce your site's visibility for user-directed web search."

Claude-SearchBot decides whether you are in the candidate pool

Claude-SearchBot indexes content so Claude knows your page exists before anyone asks. Block it and you are not excluded from one answer, you are excluded from consideration.

User agent	Job	Can it cite you?	Cost of blocking it
ClaudeBot	Collects training data for future Claude models	No	Your content is left out of future training sets. No effect on live citations.
Claude-User	Fetches a page during a live Claude conversation	Yes, directly	You disappear at the moment a buyer asks Claude about your category.
Claude-SearchBot	Builds the search index behind Claude answers	Yes, indirectly	You are dropped from the candidate pool before the question is even asked.

The OpenAI fleet has the same three-way shape, and teams get it wrong in the same way. We wrote that one up in is ChatGPT-User allowed in your robots.txt.

Should you block ClaudeBot?

Block ClaudeBot if bandwidth is a real cost or you have a policy position on training data. Allow it if you want your brand inside Claude's trained knowledge. Either way, the decision is smaller than it feels: ClaudeBot cannot cite you, so blocking it does not cost you a single AI citation. The bots that matter for visibility are Claude-User and Claude-SearchBot.

What blocking ClaudeBot actually stops:

• •Your future pages entering Anthropic's training corpus
• •The bandwidth cost of a bot that fetched us 1,129 times a day at peak
• •Nothing else

What blocking ClaudeBot does not stop:

• •Claude citing you today, which runs through Claude-User
• •Claude knowing your page exists, which runs through Claude-SearchBot
• •Content Anthropic already collected before you changed the file
• •Traffic from any other AI crawler, each of which needs its own directive

The honest framing is that this is a policy choice with a small bandwidth dividend, not a visibility lever. The visibility lever is two lines further down in the same file.

The one number that argues for allowing it

Cloudflare measured Anthropic's crawl-to-refer ratio at nearly 71,000 page requests per referral during the week of June 19 to 26, 2025. That figure is why publishers started blocking. It has also improved sharply since, and it was always a ratio about referral clicks, not about being named in an answer.

Our own 63-day study of 90,132 AI answers found that answer engines cite sources constantly: ChatGPT named a source in 92.5% of answers and Google AI Mode in 97.4%. The full set is on our AI search statistics page. Getting into that pool is the game. A training crawler is not how you get there, but it is also not the thing costing you.

How to set your robots.txt for Anthropic's crawlers

Five steps, in the order that matters. Do these once and the question stops recurring.

Step 1: Name all three agents separately

There is no wildcard that covers Anthropic's fleet. Each user agent needs its own block, or the rule silently applies to one bot and not the others.

Step 2: Allow Claude-User and Claude-SearchBot before you touch anything else

These are the two that produce and support citations. If your current file blocks either, that is today's fix and it outranks every content project on your list.

Step 3: Make the ClaudeBot call deliberately

Allow it if you want the training exposure. Disallow it if bandwidth or policy says otherwise. If you want a middle path, Anthropic's documentation supports Crawl-delay: 1 for ClaudeBot instead of a full block.

User-agent: ClaudeBot
Crawl-delay: 1

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Step 4: Verify the bots by IP, not by user agent string

Pull the prefixes from claude.com/crawling/bots.json and check your logged requests against them. Our fake Claude-SearchBot hit would have failed that check instantly. Anthropic's own guidance warns that IP blocking is unreliable as a control, but IP verification is still the right way to confirm identity.

Step 5: Publish a Markdown alternate and let the crawl take it

Our logs say ClaudeBot fetched the .md version of every HTML page it took. A clean Markdown twin, declared in the page head, gives every Anthropic bot a version with no navigation, no scripts, and no layout noise between it and your answer. It is the same logic as writing for passage extraction rather than whole pages. This is a build task, not a content task, and it applies site-wide once.

If you would rather not run this yourself, a managed GEO agency can own the crawler policy, the log verification, and the page structure work as one workstream.

FAQ

What is ClaudeBot?

ClaudeBot is the web crawler Anthropic uses to collect public content for training Claude models. It makes standard HTTP GET requests, respects robots.txt directives, and does not send referral traffic or produce citations in Claude's answers.

What is the ClaudeBot user agent string?

The string we log is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). It was identical in every sample we checked, from the first day of the window to the last. Claude-User and Claude-SearchBot use separate strings with their own contact URLs.

How do I block ClaudeBot in robots.txt?

Add User-agent: ClaudeBot followed by Disallow: /. That directive applies only to ClaudeBot. Claude-User and Claude-SearchBot each need their own entry, and blocking those two is what removes you from Claude's live answers.

What are the ClaudeBot IP ranges?

Anthropic publishes authorised crawler prefixes as a JSON file at claude.com/crawling/bots.json. The version we checked listed 26 prefixes across Google Cloud, Azure, and AWS ranges. Check logged requests against that list before trusting the user agent.

Does blocking ClaudeBot stop Claude from citing my site?

No. Citations come from Claude-User and Claude-SearchBot. In our logs ClaudeBot outnumbered Claude-User 617 to 1, and none of those 106,107 requests could produce a citation. Blocking ClaudeBot removes you from training data only.

Where to start this week

Open your robots.txt and search for Claude. Most sites we audit find either nothing, which is fine, or a blanket block written in 2024 when there was only one bot to block, which is not.

Then pull last month's logs, filter for the three Anthropic user agents, verify them against the published IP prefixes, and count how many requests were Claude-User. That number is your real Anthropic visibility signal. Ours was 172, and the 106,107 sitting next to it told us nothing about whether Claude will name us tomorrow.