Configure robots.txt for AI Crawlers in WordPress

To block AI crawlers in WordPress, edit the robots.txt file in your site root to disallow specific User-Agents like GPTBot and ClaudeBot. This restricts their access to your content without altering how standard search engines index your pages.

Most guides treat bot blocking as a blanket performance issue, ignoring the distinct User-Agent strings for individual AI models. They also fail to clarify the exact file path or the syntax required to allow Googlebot while denying training bots. Without precise configuration, you risk accidentally de-indexing your site or missing the correct verification methods for AI-specific crawlers.

Identify which AI bots visit your site

Server Logs distinguish GPTBot, ClaudeBot, and Googlebot by their unique User-Agent strings. Before configuring robots.txt, you must verify which agents are actually accessing your WordPress site. Standard analytics tools often group these requests into “Other” or “Unknown” traffic, obscuring the specific bot identity. To see the truth, access your web server logs directly via FTP or your hosting control panel. Look for the access.log file in your site’s root directory or the /logs/ folder if your host separates them.

Search the log file for the following User-Agent identifiers:

• GPTBot: Appears as GPTBot/1.0 or similar. This is OpenAI’s crawler, used to train their language models.
• ClaudeBot: Appears as ClaudeBot/1.0. This is Anthropic’s crawler, serving a similar training purpose.
• Googlebot: Appears as Googlebot/2.1. This is Google’s indexing crawler, essential for standard SEO visibility.

The critical difference is intent. Googlebot indexes your content for search results, while GPTBot and ClaudeBot scrape it for machine learning training. Blocking the latter does not affect your rankings in Google, but it prevents your text from being ingested into AI models. If you see high volumes of GPTBot traffic in your Server Logs, you have the data needed to justify blocking them. If you are unsure how your site’s structure impacts these crawlers, review our Checklist: Making Your WordPress Site Readable by AI. Understanding the raw traffic data is the foundation for the optimisation strategies in Optimising WordPress for AI Search: A Practical Guide. Once you have confirmed which bots are visiting, you can proceed to the specific blocking rules.

Locate the robots.txt file in WordPress

robots.txt file resides in the root directory of your WordPress installation, typically at /public_html/robots.txt or /var/www/html/robots.txt. You cannot edit this file directly within the WP-admin dashboard. Access requires server-level tools. To locate and edit the file, use one of these methods:

1. cPanel File Manager: Log in to your hosting control panel. Navigate to “File Manager.” Select the domain root. Find robots.txt. Right-click and select “Edit.”
2. FTP Client: Connect using FileZilla or Cyberduck. Navigate to the public root folder. Download robots.txt. Edit locally in a text editor. Re-upload the file, overwriting the original.
3. WP-CLI: If your server supports command-line access, use cat robots.txt to view the content. Use echo "new rule" >> robots.txt to append lines, though manual editing via FTP is safer for complex syntax.

If the file does not exist, create it. In cPanel, click “New File.” Name it robots.txt. In FTP, upload a new text file with that name. Ensure the file permissions are set to 644 to allow the web server to read it. For sites deployed in containerised environments, the file location may differ. If you are running WordPress on Kubernetes, the file might be mounted from a ConfigMap rather than the filesystem. Check your deployment manifests. We have detailed how to manage configuration files in containerised setups in our guide on How to Deploy WordPress with Redis Cache and MySQL on Kubernetes: Complete Step-by-Step Tutorial. Verify the file is accessible by visiting yourdomain.com/robots.txt in a browser. If you see the file contents, the server is serving it correctly. If you receive a 404 error, the file is missing or the path is incorrect.

Writing rules to allow or block specific agents

robots.txt rules target specific AI agents using the User-Agent directive paired with Disallow or Allow commands. To block GPTBot while permitting ClaudeBot, define separate rule blocks for each agent. The order of your blocks does not matter, but each block must be self-contained. Standard search engines like Googlebot and Bingbot remain unaffected because you are not writing rules for their user agents. This approach preserves your SEO visibility while controlling access for specific AI models. If you want to expose site actions to these agents instead, see WebMCP: How to Expose Site Actions to AI Agents.

User-agent: GPTBot
Disallow: / User-agent: ClaudeBot
Allow: /

The Disallow directive with a slash blocks the entire site for that specific agent. The Allow directive with a slash explicitly permits access. You can combine these to create granular access levels. For example, if you want to block GPTBot from your product pages but allow it to view your blog, you would write:

User-agent: GPTBot
Disallow: /products/
Allow: /blog/

Within a single User-Agent block, the most specific rule wins. If a URL matches both an Allow and a Disallow rule, the Allow rule takes precedence. This is critical when you are trying to block most of your site but keep certain sections open. Always test your rules after saving. A single typo in a user agent name, such as writing GPTbot instead of GPTBot, will cause the rule to be ignored entirely. Use the exact casing provided by the AI provider’s documentation. If you are unsure whether a rule is working, check your server logs for requests from that specific user agent to confirm the behaviour matches your intent.

Testing visibility with AI crawler simulators

Verifying that your robots.txt rules actually restrict AI crawler access requires testing with the specific user-agent strings those bots identify themselves by. Standard browser checks won’t work because they send a different user-agent. You need to simulate the exact request an AI agent makes.

Use cURL in your terminal to fetch your robots.txt file while spoofing the user-agent. This tells you exactly what the server sends back for that specific bot. Run this command, replacing yourdomain.com and the user-agent string:

curl -A "GPTBot" https://yourdomain.com/robots.txt

If your rule blocks GPTBot, the response should include Disallow: / for that agent. If you see Allow: / or no specific rule, the bot can still crawl. Repeat this test for each agent you configured, such as ClaudeBot or PerplexityBot. The response must match your intended rule for every single agent.

For a visual check, use an AI Crawler Simulator tool. These web-based utilities let you paste your URL and select a specific bot type. They render the effective rules after parsing your file, showing you exactly which paths are accessible. This is useful for catching syntax errors that might cause the entire file to be ignored by some parsers. If the simulator shows the wrong behaviour, check for missing line breaks or incorrect indentation in your robots.txt file.

Monitoring ongoing traffic is just as important as the initial test. In your WordPress analytics or server logs, filter requests by user-agent. Look for spikes in traffic from GPTBot or ClaudeBot after you deploy your changes. If you see continued high-volume requests from blocked agents, your rule may not be applied correctly, or the bot may be ignoring the directive. Some AI crawlers do not respect robots.txt, so if you need strict control, consider IP-based blocking in your firewall or .htaccess file. For teams integrating AI tools into their site build process, understanding how these agents interact with your content structure is part of the broader workflow described in our Connect Elementor to AI via MCP: Setup Guide | CloudyWP.

Monitoring server load after changes

Monitoring server load after changes confirms whether your robots.txt rules impact resource usage. Blocking agents like PerplexityBot reduces concurrent connections, potentially lowering CPU and memory consumption during peak traffic. To verify this, check your hosting dashboard for resource utilisation graphs over the 48 hours following deployment. Look for dips in CPU spikes that previously correlated with AI crawler surges. If you use a WordPress analytics plugin, filter traffic sources to isolate bot activity. Compare page load times before and after the change. A reduction in server response time indicates that fewer non-human requests are competing for bandwidth. If load remains unchanged, the AI traffic volume on your site is negligible, and the primary benefit is content control rather than performance. Ensure your monitoring tools distinguish between human and bot traffic to avoid misinterpreting data. If you see increased errors or timeouts, revert the rules and investigate server capacity. Regular monitoring ensures your optimisation strategy aligns with actual infrastructure limits. For broader workflow improvements that reduce manual overhead, consider our WordPress Automation services. Understanding how AI tools interact with your content structure is also vital; see How to Connect AI Tools to WordPress Using MCP for integration details. Blocking AI crawlers does not directly affect Google rankings, as Googlebot is a separate agent. However, maintaining optimal server performance indirectly supports SEO by ensuring fast load times for all users. Keep your monitoring routine consistent to catch any unexpected behaviour early.

Frequently asked questions

How to block GPTBot in WordPress robots.txt

Add the specific user-agent line for GPTBot followed by a Disallow directive in your robots.txt file. This tells the crawler it cannot access any part of your site. You must save the file to the root directory for the change to take effect immediately. Ensure you do not accidentally block Googlebot or Bingbot in the same block, as that would harm your standard search visibility.

Where is robots.txt located in WordPress

The file resides in the root directory of your web server, which is typically the public_html or htdocs folder. It is not stored within the wp-admin area or the database. You can access it via FTP, your hosting control panel’s file manager, or a file manager plugin if you have one installed. Always create a backup copy before editing to restore the original if you make a syntax error.

Does blocking AI crawlers affect Google rankings

No, blocking specific AI agents like GPTBot or ClaudeBot does not affect your Google rankings. Google uses its own crawler, Googlebot, which is a separate entity from these AI bots. As long as your robots.txt file explicitly allows Googlebot to access your content, your SEO performance remains unchanged. Blocking third-party AI crawlers only restricts their access to your data for training purposes.

How to allow ClaudeBot but block others

List the user-agent for ClaudeBot with an Allow directive, then list the other agents you wish to block with Disallow directives. The order of these rules matters in some parsers, so keep the specific allows at the top. For example, you can specify User-agent: ClaudeBot followed by Allow: /, then User-agent: GPTBot followed by Disallow: /. This granular control lets you decide exactly which AI models can access your site content.

How to test if robots.txt is working for AI bots

Use the URL of your robots.txt file in a browser to verify the syntax is correct and visible. To test specific AI bot behaviour, you can use online robots.txt checkers that simulate different user-agents. Some services allow you to input a specific bot name to see if the rules apply correctly. You can also check your server logs to see if the specific bot is still attempting to access blocked pages, which indicates the rule is not being respected.

Can I edit robots.txt in wp-admin

Not directly, unless you install a plugin specifically designed for managing robots.txt files. The native WordPress dashboard does not have a built-in interface for editing this server-level file. Most developers prefer using FTP or the file manager in their hosting control panel for direct access. If you use a plugin, ensure it is reputable and regularly updated to avoid conflicts with other SEO tools.

What to do next

Save your current robots.txt to your local machine before you close the browser tab. This single file is the only control point between your site and every crawler that visits it. If you make a mistake in the syntax, you can restore the previous state in seconds rather than waiting for search engines to re-crawl your entire site. Keep a plain text copy in a dedicated folder on your desktop, labelled with the date of the change. This habit prevents the most common error we see: a developer edits the file, breaks a rule, and has to guess what the original looked like. The file is small, but its impact is site-wide. Treat it with the same care you would treat your database dump. If you are unsure whether a rule is working, use the testing tools mentioned in the steps above to verify visibility before you move on. Do not rely on guesswork. The rules are explicit, and the feedback is immediate. You have the syntax, the location, and the verification method. The next step is to apply the changes to your live site and monitor the server logs for the next 48 hours to confirm the behaviour matches your intent.