AI SEO Glossary

Robots.txt is a text file placed at the root of a website that tells web crawlers which pages or sections they are allowed to access. Originally designed for search engine bots like Googlebot, it has taken on new significance as AI companies deploy their own crawlers to collect web content for training and retrieval. You can use robots.txt to explicitly allow or block crawlers like GPTBot, ClaudeBot, and PerplexityBot from accessing your site. The decision is not straightforward. Blocking a crawler may prevent your content from appearing in that platform’s AI responses, while allowing it means your content may be used in training data and real-time retrieval.

Blocking or allowing specific AI crawlers in robots.txt follows the same syntax as managing any other crawler. To block GPTBot from your entire site, you add a User-agent: GPTBot directive followed by Disallow: /. To allow it while blocking it from specific directories, you use Disallow: /private/ or similar path-level rules. Most AI companies publish their official crawler user agent strings in their documentation, so you can configure rules for each individually. Before making changes, consider the tradeoff: blocking an AI crawler protects your content from training use but also reduces your visibility in that platform’s search and answer features.

Why it matters: Robots.txt is now a strategic decision about which AI platforms you want to appear in. Most site owners have not yet made that decision intentionally.