AI crawlers, robots.txt & llms.txt

Lesson 7 of 8 · 30 min · part of the AI SEO & GEO Specialist certification

AI companies run several different bots with different jobs. The key distinction: bots that collect data for training models, bots that index pages for AI search, and agents that fetch a page because a user asked right now. You can allow one and block another in robots.txt.

Interactive exercise: Match the user-agent to its job

Anthropic also uses Claude-User for fetches a user asks for. Perplexity uses PerplexityBot for its index (and Perplexity-User for user-requested fetches).
Google-Extended does not affect Google Search — blocking it does not remove you from Search or from AI Overviews, which are part of Search and use Googlebot.
Microsoft Copilot answers are grounded in the Bing index, crawled by Bingbot.
Bots that act on a direct user request may not follow robots.txt the way crawlers do — check each vendor's documentation.

Allow AI search, opt out of model training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Interactive exercise: The publisher's dilemma