The question comes up in every ops chat in 2026: should we let GPTBot crawl the site, or block it in robots.txt? A lot of teams blocked it in a panic in late 2023, then quietly unblocked it a year later once ChatGPT started sending referral traffic. Others still block, on principle. Both camps have valid reasons and neither answer is universal.
This is the honest decision framework: what each of OpenAI's three crawlers actually does, what you gain and lose in each direction, a matrix by business type, and copy-paste snippets for both outcomes.
OpenAI ships three separate bots. They are not the same decision.
Lumping them together is the first mistake. As of 2026, OpenAI publishes three distinct User-Agents, each with a different job:
GPTBot— the training crawler. Pulls pages into the corpus that trains future GPT models. Does not directly send you traffic.ChatGPT-User— the live browse agent. Fires when a ChatGPT user explicitly asks the model to look something up on your site. Sends citation links back into the ChatGPT UI.OAI-SearchBot— the SearchGPT index crawler. Populates the ChatGPT search surface (the one that competes with Google). Sends citation traffic.
You can block one and allow the others. Most teams that think they want to "block OpenAI" actually want to block only GPTBot (the training one) while letting ChatGPT-User and OAI-SearchBot through so their site is still findable in ChatGPT answers. Doing all three at once is a common accident that quietly disappears you from ChatGPT citations.
What you lose by blocking
- Block
GPTBot: your content is not used to train future OpenAI models. In practice, this rarely moves the needle for one site among billions, but the opt-out is real and enforced. - Block
ChatGPT-User: when a ChatGPT user types "summarise this page" or asks a question that pivots on your URL, the model gets a fetch error and moves on. You lose direct citation traffic that was already showing intent. - Block
OAI-SearchBot: you disappear from SearchGPT results. Growing surface, real referral clicks in 2026.
What you gain by blocking
- Bandwidth relief. Real if you serve heavy pages (large product catalogs, media archives). Negligible for most brochure sites.
- Copyright hygiene. If your content is licensed (paywalled journalism, premium research, paid courses), blocking training-tier crawlers is a defensible legal signal, especially in the EU under the TDM opt-out framework.
- Competitive moat. If your data is your product (pricing databases, proprietary datasets, benchmark tables), you probably do not want it in the training set of every model.
Decision matrix by business type
| Business type | GPTBot (training) | ChatGPT-User (live browse) | OAI-SearchBot (index) |
|---|---|---|---|
| SaaS marketing site, docs, blog | Allow | Allow | Allow |
| Ecommerce (public product pages) | Allow | Allow | Allow |
| News / paywalled journalism | Block | Allow (or block if paywall covers previews) | Allow |
| Paid research / data-as-a-product | Block | Block | Allow (index only titles) |
| Community forum with user-generated content | Depends on ToS with users | Allow | Allow |
| Personal portfolio, agency, brand site | Allow | Allow | Allow |
Default answer for a growth-stage site with no unique data moat: allow all three. You want to be cited, referenced, and browsed. Blocking is the choice when your content itself is the product you sell.
How to allow (recommended default)
Do nothing in robots.txt. OpenAI crawlers respect standard Allow/Disallow rules, and the default (no rule) is allow. But if you want to be explicit (which helps auditors and AI-readiness scanners like agentfix.pro), publish this:
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /Explicit allow rules are also useful when another rule higher in your file (like WordPress' default Disallow: /?) would otherwise apply. See the full AI crawler robots.txt guide for the complete 2026 list.
How to block (if you decided to)
Split the decision by bot. Block only what you mean to block:
# Block training only (typical opt-out)
User-agent: GPTBot
Disallow: /
# Keep live browse and SearchGPT allowed
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /If you want a full lockout instead:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /Verify the decision took effect
curl -A "GPTBot" https://yoursite.com/robots.txt— sanity check that the file serves.- Fetch your homepage with each User-Agent header and confirm the response you expect (200 if allowed, or that your CDN honors the block).
- Ask ChatGPT to "visit yoursite.com and summarise the homepage". If browse is allowed and works, you get a summary. If blocked, you get a fetch error message.
- Run a scan on agentfix.pro — the report flags conflicting rules (a common bug is
Disallow: /under a wildcard that accidentally covers OpenAI bots too).
Related reading
- The complete 2026 AI crawler robots.txt list — every bot, not just OpenAI.
- The 33-signal GEO checklist — where robots.txt sits in the wider AI-readiness picture.
- Schema.org JSON-LD for AI search — the layer above robots that decides how you get cited.
- The /.well-known directory for AI agents — the layer next to robots.
DIY or fix-pack
If you have shell access and one homepage, this is a five-minute copy-paste. Do it yourself. If you run a large site, have redirects and legacy rules, or want an auditor-grade explanation of which bots you allow and why (for a compliance file), the $29 AgentFix pack ships a robots.txt tuned to your live scan plus a decision log you can hand to a stakeholder. Either way, the important part is that you made an informed choice per bot, not a reflex block of everything with "GPT" in the name.