Free Tool
Robots.txt Generator
Build a robots.txt with one-click AI-crawler blocking — GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and more — plus modern search-bot presets, CMS templates, and live syntax checks.
Default access (User-agent: *)
Disallowed paths
AI & LLM crawlers
GPTBotOpenAI · Trains ChatGPT models
ChatGPT-UserOpenAI · ChatGPT live browsing on user request
OAI-SearchBotOpenAI · ChatGPT Search indexing
ClaudeBotAnthropic · Trains Claude models
Claude-SearchBotAnthropic · Claude search results
Claude-UserAnthropic · Claude live browsing on user request
anthropic-aiAnthropic · Legacy Anthropic crawler
Google-ExtendedGoogle · Gemini / Vertex AI training (does not affect Search)
CCBotCommon Crawl · Open dataset used to train many LLMs
PerplexityBotPerplexity · Perplexity answer index
Perplexity-UserPerplexity · Perplexity live browsing on user request
BytespiderByteDance · TikTok / Doubao AI training
AmazonbotAmazon · Alexa / Amazon AI
Applebot-ExtendedApple · Apple Intelligence training (does not affect Siri/Spotlight)
Meta-ExternalAgentMeta · Meta AI training
cohere-aiCohere · Cohere model training
DiffbotDiffbot · Knowledge-graph extraction
TimpibotTimpi · Decentralized search/AI index
Search engine crawlers
GooglebotGoogle · Google Search
Googlebot-ImageGoogle · Google Images
Googlebot-NewsGoogle · Google News
AdsBot-GoogleGoogle · Google Ads landing-page quality
BingbotMicrosoft · Bing Search (also powers Copilot)
DuckDuckBotDuckDuckGo · DuckDuckGo Search
YandexBotYandex · Yandex Search
BaiduspiderBaidu · Baidu Search
SlurpYahoo · Yahoo Search
ApplebotApple · Siri & Spotlight Suggestions
Sitemap URL(s)
- No sitemap added — including your sitemap URL helps crawlers discover all pages.
Control crawlers — and AI training — in one file
Most robots.txt generators still ship dead presets (DMOZ, MSN PicSearch) and ignore the crawlers that actually matter in 2026: the AI bots harvesting content for training and answer engines. This generator puts the full modern list front and center — GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider, Applebot-Extended and more — so you can block AI training with one click while keeping Googlebot, Bingbot and other search crawlers fully allowed. It also flags dangerous rules (like an accidental Disallow: /) before you publish.