What Is ClaudeBot? Anthropic's Web Crawler
ClaudeBot is the web crawler operated by Anthropic, the company behind the Claude family of AI assistants. It requests publicly available web pages, and if you run a website you may find ClaudeBot among the user agents in your access logs. This page explains what it does and how to control it.
What ClaudeBot is for
ClaudeBot is Anthropic's crawler for collecting publicly available web content. Like other AI crawlers, it reads pages and follows links so that content can inform the models that power Claude. Anthropic's documentation has referenced both the ClaudeBot token and an anthropic-ai user agent, so when you write robots.txt rules it is worth accounting for the names Anthropic currently lists.
As of mid-2026, Anthropic states that its crawling respects robots.txt directives and publishes documentation to help site owners identify and control the agent. Because the AI crawler landscape is still evolving, treat the official Anthropic support documentation as the source of truth for exact token names and any IP ranges you use for verification.
Crawling vs. user-triggered fetching
It helps to distinguish two kinds of activity. Bulk crawling is when a crawler like ClaudeBot systematically reads many pages to build a broad picture of the web. User-triggered fetching is when the assistant retrieves a specific page because a person asked it to — for example, pasting a link and asking Claude to summarize it. Anthropic references a separate agent for that user-initiated behavior. The two are governed by different tokens, which means you can make different decisions about each.
Allowing or blocking ClaudeBot
Control lives in robots.txt at your domain root. To disallow ClaudeBot completely:
User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: /
To allow it while keeping a private area off-limits:
User-agent: ClaudeBot Allow: / Disallow: /account/
These directives express a preference to a cooperating crawler. robots.txt cannot technically force compliance, so for strict control, block at the server level. And keep in mind that llms.txt is advisory only — it curates content for language models and does not block ClaudeBot or any other crawler.
Should you allow ClaudeBot?
The decision mirrors the one for other AI crawlers. Allowing ClaudeBot gives Anthropic's models more of your content to draw on, which can help Claude describe your business or cite your expertise accurately. Blocking it keeps your content out of that pipeline but can reduce your representation in AI answers over time. Publishers protecting premium content often restrict crawling; businesses focused on discovery frequently allow it. Choose based on whether your priority is control or visibility.
Frequently Asked Questions
What is the ClaudeBot user agent?
The robots.txt token is ClaudeBot. Anthropic has also referenced an anthropic-ai user agent in its documentation. Check Anthropic's official crawler support pages for the current, authoritative token list and any published IP ranges.
What does ClaudeBot collect?
ClaudeBot is described as Anthropic's web crawler for gathering publicly available content. Anthropic separately references a user-triggered agent for fetches Claude makes on a person's behalf, which is a different activity than bulk crawling.
Does ClaudeBot respect robots.txt?
Anthropic documents that its crawler respects robots.txt. As with all crawlers, robots.txt is advisory rather than enforceable, so it functions as a respected signal to a cooperating operator, not a technical guarantee.
How do I block ClaudeBot?
Add a robots.txt rule naming ClaudeBot with Disallow: /. If Anthropic lists additional tokens such as anthropic-ai, add a matching block for each token you want to exclude.
Will blocking ClaudeBot affect Claude's answers about my business?
It may. If Anthropic's models cannot access your content, they have less material to represent your business accurately. That is the visibility tradeoff to weigh against your reasons for restricting crawling.
See how AI understands your business
Generate an llms.txt file to help AI assistants like Claude read and represent your content accurately.