AI Training vs. AI Search: Setting Up Your WordPress robots.txt for Each

There is a quiet setting on your WordPress site that might be sending away your best future customers. It lives in a small text file called robots.txt, and most small business owners never look at it. The trouble is that a security plugin, a firewall, a CDN, or a default SEO configuration may have already decided to block the AI crawlers that could be putting your business in front of buyers.

Here is the part that trips people up. There are two very different kinds of AI bots visiting your site, and your WordPress robots.txt often treats them as the same thing. One group collects your content to train large language models. The other group fetches your pages in real time to answer a customer’s question and cite you as the source. Blocking the first is a reasonable choice. Blocking the second can quietly cost you traffic that tends to convert well.

This guide walks through the distinction, shows you how WordPress generates its robots.txt file, and gives you a safe setup that opts your content out of training while keeping you eligible for AI search citations.

Why your WordPress robots.txt matters more now

AI search is no longer a side experiment. People ask ChatGPT, Perplexity, Claude, and Google’s Gemini for recommendations, and those tools pull answers from live web pages. If a retrieval bot can’t reach your pages, your business can’t be cited in the answer.

The traffic that comes back from these answer engines also behaves differently from a casual search click. A 2025 Semrush study estimated that, measured by conversion rate, the average AI search visitor is worth about 4.4 times as much as the average organic search visitor. And in April 2026, Adobe reported that AI-referred traffic to US retail sites in the first quarter of 2026 was up 393 percent from a year earlier.

In Semrush’s 2025 analysis, the average AI search visitor was worth about 4.4 times as much as an average organic visitor by conversion rate. A blanket robots.txt block can shut out some of your highest-intent prospects.

Those are the people you risk losing when a blanket rule in your WordPress robots.txt blocks everything with “AI” in its name. The fix isn’t to open the floodgates. It’s to understand which bot does what.

AI training crawlers vs. AI search crawlers

The single most useful thing you can learn here is that AI companies run separate bots for separate jobs. A training crawler collects text to help build or refine a model. A search or retrieval crawler fetches a page because a user just asked a question, then often links back to you.

You can allow the bots that cite you while declining the ones that only feed model training. Here are the common names you’ll see in server logs and robots.txt files:

• Training crawlers and tokens: GPTBot (OpenAI training), ClaudeBot (Anthropic training), CCBot (Common Crawl, widely used for training datasets), and Google-Extended (a robots.txt control token for Gemini, not a separate crawler).
• Search and retrieval crawlers: OAI-SearchBot and ChatGPT-User (OpenAI search and on-demand fetching), Claude-SearchBot and Claude-User (Anthropic search and user-directed fetching), and PerplexityBot (Perplexity’s answer engine).

One nuance on Google: Google says blocking Google-Extended doesn’t affect your inclusion or ranking in Google Search, but it does cover grounding in the Gemini app as well as model training. Treat that as a trade-off, not a free switch.

The lines can blur, and bots change roles over time, so it pays to check vendor documentation now and then. The principle holds regardless: you have granular control, and your WordPress robots.txt is the first place to exercise it.

The new standard for separating the two

The industry has started to formalize this split. On October 30, 2026, Anthropic, OpenAI, Google, and Microsoft announced support for a shared robots.txt signal, tied to IETF drafts, that lets a site say “don’t use this for AI training” in one place. It uses a special user-agent token called __usage_ai-training:

User-agent: __usage_ai-training
Disallow: /

Support is rolling out on different timelines. Anthropic says it honors the token today. OpenAI has said it will follow in the coming months, Microsoft plans to honor it starting in February 2027, and Google plans support in early 2027. Until all of them do, treat the token as a supplement to clear user-agent rules rather than a replacement.

How WordPress generates robots.txt

Before you edit anything, you need to know what you’re actually editing. WordPress doesn’t ship a physical robots.txt file by default. Instead it creates a virtual one on the fly whenever a bot requests it.

If you visit yoursite.com/robots.txt and see a short set of rules, that may be the virtual version generated by WordPress core. The moment you upload a real robots.txt file to your root directory, that physical file takes over and the virtual one is no longer served.

SEO plugins add another layer. Rank Math, Yoast, and similar tools can write their own robots.txt rules through their settings panels, and those rules override the core defaults. This is where many surprises hide, because a plugin update or a changed setting can alter what that file says without you noticing.

The silent block problem

Robots.txt isn’t the only place AI bots get turned away. In July 2025, Cloudflare began blocking known AI crawlers by default for new domains, unless the site owner chooses to allow them. Many security plugins, hosting firewalls, and ad-network tools now offer one-click AI crawler toggles too.

These controls are useful, but they’re easy to flip without realizing the “off” position may also turn away the search and retrieval bots that drive referrals. A clean robots.txt can’t help if a firewall rule upstream is already refusing those visitors.

The lesson is simple. Whatever plugin, CDN, or host manages your bot rules, open it and read exactly what it’s blocking. Don’t assume the defaults match your goals.

A safe robots.txt setup for AI search

Here’s a practical configuration that opts out of training while staying visible to AI search. You can adapt the exact bots to your comfort level, but this structure keeps the retrieval crawlers welcome.

The approach follows three moves:

1. Block the training crawlers you don’t want feeding model datasets.
2. Explicitly allow the search and retrieval crawlers that cite sources.
3. Leave your standard search engines (Googlebot, Bingbot) fully open so your traditional SEO index is untouched.

A sample set of rules looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

Keep your existing WordPress rules (such as the default block on /wp-admin/) in place alongside these. If you’d rather keep Gemini grounding, leave out the Google-Extended rule.

If you prefer a guided route over hand-editing, the Better Robots.txt plugin on WordPress.org gives you a settings interface for managing these rules and lets you preview the file before it goes live. Whatever tool you use, verify what it writes after each update rather than trusting preset toggles. For a broader walkthrough of what to allow, block, and confirm, this WordPress robots.txt guide from StackHarbor covers the verification steps in detail.

Add the training preference token

If you want to layer in the shared standard, add the __usage_ai-training group shown earlier alongside your user-agent rules. Because support is still rolling out, keep your explicit GPTBot, ClaudeBot, and Google-Extended rules as the dependable backstop. Think of the token as a forward-looking signal, not your only defense.

Checking and testing your live file

Editing robots.txt without verifying the result is how sites stay broken for months. Once you make changes, confirm that the live file says what you intended.

Run through these checks in order:

1. Visit the file directly. Go to yoursite.com/robots.txt in a browser and read every line. This shows you the version the world actually sees, virtual or physical.
2. Check Google Search Console. Use the robots.txt report to confirm Google fetched your latest file without errors, and spot-check that your important pages can still be crawled.
3. Check your server logs. Look for the bots you allowed, such as OAI-SearchBot or PerplexityBot, to confirm they’re reaching your pages and not being refused by a firewall.
4. Re-check after every plugin update. Since SEO, security, and ad plugins can rewrite rules, make this a recurring habit rather than a one-time task.

If the live file doesn’t match your settings panel, a physical robots.txt file in your root directory is probably overriding the plugin. Remove or edit that physical file to let your chosen tool take control.

What robots.txt cannot do

It helps to be honest about the limits here, because overconfidence in a single file leads to bad decisions. Your WordPress robots.txt is a request, not a wall.

Keep these constraints in mind:

• Compliance is voluntary. Well-behaved bots follow robots.txt, but aggressive scrapers can ignore it entirely. The Robots Exclusion Protocol (RFC 9309) defines the rules, yet it can’t force anyone to obey them.
• Crawling is not indexing. Disallowing a crawler stops it from reading the page, but it doesn’t remove an already-indexed URL. For that, you need a noindex directive, which is a separate control.
• Allowing bots doesn’t guarantee citations. Letting OAI-SearchBot in makes you eligible to appear in answers. Whether you actually get cited depends on your content quality, structure, and authority.

For scrapers that disregard the file, you need another layer. Hosting-level firewall rules and a CDN such as Cloudflare, which offers AI crawler controls, give you enforcement that robots.txt alone can’t provide. Robots.txt handles the cooperative bots; the firewall handles the ones that don’t play fair.

Turning crawl access into actual visibility

Getting your WordPress robots.txt right is the entry ticket, not the whole show. Once the right crawlers can reach you, the next job is giving them content worth citing. AI systems favor clear structure, direct answers, and strong trust signals.

A few practical priorities follow naturally from here:

• Structure pages with clean semantic HTML and descriptive headings so retrieval bots can parse your meaning quickly.
• Add schema markup that tells AI systems who you are, what you offer, and where you operate.
• Answer real customer questions directly near the top of the page, since answer engines reward concise, self-contained responses.

If you want the full picture of how these pieces fit together, our complete WordPress guide to optimizing for AI search walks through structure, schema, and trust signals step by step. For a shorter primer on the core concepts, this overview of how to optimize for AI search covers the essentials without the jargon.

Putting it all together

The core idea is worth repeating because it’s easy to miss. AI training crawlers and AI search crawlers do different jobs, and your WordPress robots.txt can treat them differently. You can decline to feed model training while staying eligible for the AI answers that send high-intent visitors your way.

Start with an audit. Open your robots.txt, read what your plugins have written, check your CDN and firewall settings, and compare it all against your goals. Block the training bots you want to exclude, welcome the search bots that cite sources, and leave your standard Google and Bing crawling intact. Then verify the live file and check it again after every plugin update, since silent overrides are the main way good setups quietly break.

At AgenticPress, we build and tune WordPress sites so they perform in both traditional search and AI answer engines, which means getting these crawl rules right is part of the groundwork, not an afterthought. If you’re unsure whether your current setup is quietly turning away AI search traffic, audit that file today and fix anything that blocks the bots you actually want. A few minutes in your robots.txt could be the difference between appearing in an AI answer and being left out of it entirely.