Half of the AI-discoverability problem in 2026 is sites that quietly block AI crawlers in robots.txt without realising it. The default WordPress robots.txt is the worst offender — it disallows /wp-admin/ (fine) but also blocks /? query strings (which silently kill query-parameterised product URLs from crawl).
This is the complete 2026 list of AI crawler User-Agents, what they do, and a copy-paste robots.txt that welcomes them properly.
The 2026 AI crawler list
These are the User-Agents AI agents and AI-search products use as of mid-2026:
| User-Agent | Owner | Purpose |
|---|---|---|
GPTBot | OpenAI | Training data + ChatGPT browse fetches |
ChatGPT-User | OpenAI | Live ChatGPT browse on user request |
OAI-SearchBot | OpenAI | SearchGPT index |
ClaudeBot | Anthropic | Training + Claude browse |
Claude-Web | Anthropic | Live Claude browse |
anthropic-ai | Anthropic | Legacy Anthropic crawler |
PerplexityBot | Perplexity | Perplexity index |
Perplexity-User | Perplexity | Live Perplexity browse |
Google-Extended | Gemini + Vertex AI training | |
Googlebot | Standard search (still relevant — AI Overviews use it) | |
Applebot | Apple | Spotlight, Siri, Apple Intelligence |
Applebot-Extended | Apple | Apple Intelligence training opt-in |
Bytespider | ByteDance | Doubao, TikTok Search |
CCBot | Common Crawl | Open dataset used by most LLM training |
cohere-ai | Cohere | Coral / Command training |
Diffbot | Diffbot | Knowledge Graph crawler |
DuckAssistBot | DuckDuckGo | DuckAssist (Anthropic-powered) |
FacebookBot | Meta | Llama training opt-in signal |
PetalBot | Huawei | Petal Search |
YandexBot | Yandex | Yandex Search + YandexGPT |
The recommended 2026 robots.txt for most sites
This explicitly welcomes every AI crawler above and keeps your admin / sensitive paths protected:
# robots.txt for an AI-readable public site (2026)
# Welcomes AI agents, blocks admin + abuse paths.
User-agent: *
Allow: /
# Explicit AI crawler welcome list — redundant with User-agent: *
# but signals intent clearly to operators auditing your file.
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: Bytespider
Allow: /
User-agent: CCBot
Allow: /
User-agent: cohere-ai
Allow: /
User-agent: Diffbot
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: FacebookBot
Allow: /
User-agent: PetalBot
Allow: /
# Block admin + sensitive paths from ALL crawlers
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /admin/
Disallow: /.git/
Disallow: /.env
Disallow: /api/internal/
Sitemap: https://yoursite.com/sitemap.xmlIf you want to opt OUT of AI training (publishers, paywalled)
If you license your content and want to keep AI agents from training on it (while still allowing browse-on-demand for citation), use this opt-out pattern:
# Block training crawlers, allow browse-on-demand
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: FacebookBot
Disallow: /
# Allow live browse for citation (note: ChatGPT-User, Claude-Web,
# Perplexity-User respect robots.txt but are fetching ONE page per
# user query, not bulk training)
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
# Keep search engines happy
User-agent: Googlebot
Allow: /
User-agent: bingbot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlThis is the publisher posture: no training, yes citation. Major news sites (NYT, FT, Reuters) ship variations of this.
WordPress-specific gotchas
- Default WordPress robots.txt is generated by core if no static file exists. It blocks
/wp-admin/(fine) but also serves a single ruleDisallow:for everyone else, which is benign — except some SEO plugins overwrite it with overly aggressive rules. - Yoast and RankMath let you edit robots.txt from the dashboard. Check what they've set — neither plugin auto-allows the AI crawler list as of 2026.
- To override the dynamic file, drop a static
robots.txtin your document root. WordPress detects it and stops serving the dynamic version.
Verify your robots.txt
# Fetch and inspect
curl -sS https://yoursite.com/robots.txt
# Validate against a specific User-Agent
curl -A "GPTBot" -sI https://yoursite.com/some-page
# Look at the response code; 200 = allowed.
# Free CLI check
npx agentfix-mini-scanner yoursite.com
# Output includes: robots.txt allows AI crawlers — PASS/FAIL with mentions listWhat robots.txt does NOT do
Three common myths:
- robots.txt does not enforce blocking. Polite crawlers respect it. Malicious ones ignore it. For real blocking, use Cloudflare's Bot Fight Mode or a WAF.
- Allowing a crawler is not consent for training. The legal status of training on robots.txt-permitted content is contested. If you have strong opinions, add a
noindex+ license header too. - robots.txt is not a security boundary. Do not put sensitive paths in it — you're just listing them publicly. Use proper auth instead.
Next step
robots.txt is one signal of 33. The big leverage is in llms.txt and JSON-LD — both ship in the AgentFix Pack alongside an AI-allow robots.txt generated for your domain. From $1 one-time.