Let AI crawlers through your firewall

Many firewalls (WAFs), bot managers and security plugins block automated clients by default, so GPTBot, ClaudeBot, PerplexityBot and ChatGPT’s page fetcher get back 403 Forbidden. A blocked crawler cannot read your pages or your agent card, even when Sill is installed correctly. This page shows how Sill tells you, which rule to add, and how to check the fix.

How Sill shows a block

When Sill’s check of your homepage gets a 403, you see it in three places:

Agent readiness in the dashboard: the crawler check reads AI crawlers get 403 Forbidden. Click Show me how to open Let AI crawlers reach your site, with steps for your setup.
The All sites overview: a WAF is blocking AI agent crawlers banner listing the paths to exempt.
The readiness report email and the shared report page on sill.so show the same banner.

The banner on the All sites overview names the paths to exempt.

Fix it on Cloudflare

These are the steps the Let AI crawlers reach your site drawer gives for a Cloudflare site:

Open the Cloudflare dashboard for your site.
Go to Security → Bots and open AI Crawl Control. For each assistant you want to allow (for example ChatGPT, Claude, Perplexity, Google-Extended, Googlebot, Bingbot), set the action to Allow.
In Security → WAF → Custom rules, add a rule with Action set to Skip and this expression:
```
(cf.verified_bot_category in {"AI Crawler" "AI Assistant" "Search Engine Crawler"})
```
Save, wait a minute, then press Check again on Agent readiness.

Cloudflare decides which requests are verified bots from the network they come from, not from the user agent, so a client that only claims to be GPTBot does not match this rule.

Fix it on another firewall or host

In your host or firewall, allow the verified AI crawlers you want (ChatGPT, Claude, Perplexity, Googlebot, Bingbot). Most vendors publish the IP ranges their crawlers use.
If your WAF has verified-bot categories (Fastly, Imperva, AWS WAF, Akamai and others), add a bypass rule for the AI crawler, AI assistant and search engine crawler categories.

If your firewall can only match user agents, allow the tokens of the crawlers you want:

Crawler	User-agent token
OpenAI search	`OAI-SearchBot`
OpenAI training	`GPTBot`
ChatGPT, fetching for a user	`ChatGPT-User`
Anthropic training	`ClaudeBot`
Anthropic search	`Claude-SearchBot`
Claude, fetching for a user	`Claude-User`
Perplexity search	`PerplexityBot`
Perplexity, fetching for a user	`Perplexity-User`
Google	`Googlebot`
Microsoft Bing and Copilot	`bingbot`
Common Crawl	`CCBot`

Google-Extended is a robots.txt setting, not a user agent; Google crawls as Googlebot.

Save, wait a minute, then press Check again on Agent readiness.

Any client can send any user agent, so a user-agent rule also lets through anyone who copies it. On Cloudflare Worker and WordPress plugin installs, Sill marks visits from agents that sign their requests with Web Bot Auth as verified, so you can tell them apart from a user-agent claim.

Always allow the discovery paths

Even if you keep crawlers out of the rest of your site, allow these paths so agents can find your signed card and files. Put the rule before any block rule:

/.well-known/agent-card.json
/.well-known/mcp/server-card.json
/.well-known/ai-catalog.json
/.well-known/api-catalog
/.well-known/ucp
/.well-known/sill-proof.txt
/llms.txt
/agent
/ai-catalog.json

These are the files the Cloudflare Worker and WordPress plugin serve on your domain, and the same list the WAF is blocking AI agent crawlers banner, the report email and the shared report page show. Allowing /.well-known/sill-proof.txt also keeps domain verification working.

Check the fix

On Agent readiness, press Check again. The crawler check reads Crawlers can reach your pages. when your homepage answers 200.
Run the public readiness scan on sill.so for your domain. A scan that was blocked shows real results once the firewall lets it through.