OAI-SearchBot
OpenAI · Search
Surfaces sites in ChatGPT search. Its control is separate from GPTBot.
User-agent: OAI-SearchBot Official reference ↗Different bots.
Different jobs.
Pick the current provider token that matches your publishing policy.
Provider references reviewed 28 August 2026. Definitions change; follow the linked source before editing production rules.
GPTBot
OpenAI · Model development
Collects content that may improve OpenAI's generative foundation models.
User-agent: GPTBot Official reference ↗ChatGPT-User
OpenAI · User requested
Fetches pages for certain ChatGPT user actions. It is not an automatic crawler.
User-agent: ChatGPT-User Official reference ↗Claude-SearchBot
Anthropic · Search
Navigates the web to improve Claude search responses.
User-agent: Claude-SearchBot Official reference ↗ClaudeBot
Anthropic · Model development
Collects public content that could contribute to model training.
User-agent: ClaudeBot Official reference ↗Claude-User
Anthropic · User requested
Fetches a site after a Claude user asks the service to access it.
User-agent: Claude-User Official reference ↗PerplexityBot
Perplexity · Search
Surfaces and links sites in search results. Perplexity excludes model training from this purpose.
User-agent: PerplexityBot Official reference ↗Perplexity-User
Perplexity · User requested
Fetches pages requested by a user and generally ignores robots.txt.
User-agent: Perplexity-User Official reference ↗Google-Extended
Google · Control token
Controls certain Gemini training and grounding uses. It does not affect Google Search ranking.
User-agent: Google-Extended Official reference ↗A token is easy to spoof.
Verify important requests with provider IP data or a verified-bot service, then compare them with your access logs.
Token
The product name matched by a robots.txt policy.
Requester
The service behind a verified network request.
Outcome
Indexing, citation, recommendation, or conversion after the request.
Check your rules.
Test the exact page you care about against selected provider tokens.