Free tool
Googlebot Block Tester
We request your URL twice, once with a Googlebot user-agent and once with an ordinary browser user-agent, then show you both responses side by side. If a firewall, CDN or bot rule treats the crawler differently, the difference shows up here. No signup required.
0 URLs entered · free test covers the first 3
Requests each URL twice, changing only the user-agent, then compares status codes, response size and challenge fingerprints. No signup, no credit card.
This test is a strong lead, not proof
We send both requests from a SearchOptimo server, not from a verified Googlebot IP. Google verifies its own crawler by reverse DNS to googlebot.com or google.com plus its published IP ranges, so a firewall that allows Googlebot by verified IP will still refuse us and this test will read as blocked when Googlebot is perfectly fine. A rule that blocks on IP reputation rather than user-agent never fires for us at all, so a real block can pass this test cleanly. The authoritative check for your own site is Google Search Console URL Inspection, using Test live URL.
What this test can prove, and what it cannot
Every tool that claims to check Googlebot access from outside Google has the same limitation, and most of them do not mention it. Ours does, because the wrong answer here costs you either a security rule you needed or weeks spent looking in the wrong place.
What it can detect
- Blocking that keys on the user-agent string
- A challenge or interstitial served to the crawler agent
- Different status codes for the two agents
- Cloaking, meaning materially different content for Googlebot
- A user-agent conditional redirect
- Outright 403 or 503 responses to the crawler
What it cannot detect
- Rules that allow Googlebot by verified IP, which still refuse us
- Blocks based on IP reputation rather than user-agent
- Rate limits that only fire at real crawl volume
- Anything that happens after JavaScript rendering
- Whether Google has actually chosen to index the page
Why the limitation exists
We send from a SearchOptimo server IP carrying a Googlebot user-agent string. Anyone can send that string, which is precisely why firewalls do not trust it. Google verifies its own crawler by reverse DNS to googlebot.com or google.com plus its published IP ranges, and Cloudflare exposes the same idea as cf.client.bot, its Known Bots field. A site protected that way will refuse our request and welcome the real crawler, and this test would read as blocked when nothing is wrong. In the other direction, a rule that filters on IP reputation never fires for us, so a genuine block can pass cleanly.
Treat a result here as a strong lead worth chasing, never as a verdict. The authoritative check for a site you own is Google Search Console URL Inspection using Test live URL, because that fetch really does come from Google.
How to read your result
The verdicts differ in how much they license you to conclude. Two of them are strong evidence. One of them is mostly a statement about us rather than about your site.
| Verdict | What it means | What to do next |
|---|---|---|
| Blocked for the crawler UA | The crawler agent got a 4xx or 5xx and the browser agent did not. The user-agent was the only variable, so something is reading it and refusing. | Open your edge or WAF event log, filter for Googlebot, and find the rule that fired. Then confirm in Search Console before changing it. |
| Challenge served | The crawler agent received an interstitial rather than your page. Googlebot cannot solve a JavaScript or CAPTCHA challenge, so it would index the interstitial or record a crawl failure. | Allow verified bots to skip the challenge. On Cloudflare that is a Skip rule on Known Bots, not an allowlist on the user-agent string. |
| Status or content mismatch | Both agents got through but were treated differently. A large content difference can be cloaking, which breaks Google spam policy, or it can be a cache variant or a consent banner. | Compare the two pages yourself before concluding. Look for user-agent branching in your app, your CDN worker, or a personalisation layer. |
| Blocked for us, not for the UA | Both requests were refused. That points at our datacenter IP rather than at the user-agent, and it tells you very little about the real crawler. | Skip straight to Search Console URL Inspection. This is the case where an outside tool genuinely cannot help you. |
| No UA difference | Nothing is filtering on the user-agent. This is good news about one specific failure mode, not a clean bill of health. | If Google still is not crawling the page, look at robots.txt, indexing directives and rate limiting instead. |
If the answer turns out to be a directive rather than a network rule, the noindex checker reads robots.txt, the X-Robots-Tag header, the robots meta tag and rel=canonical in one pass. And if you want to know whether Google currently has the page at all, the Google index checker answers that separate question.
Blocking AI training can now block Googlebot
This is the newest way a site loses crawl access without anyone touching a firewall rule, and it catches people who thought they were only opting out of AI.
Cloudflare has replaced the single “block AI bots” switch with three behaviours: Search, Agent and Training. Crawlers that do more than one of those get judged by the strictest rule that applies to them. Googlebot, Bingbot and Applebot are all in that category, because each crawls for search and for AI training. So a site that blocks Training also blocks the crawler that keeps it findable. From 15 September 2026 Cloudflare applies new defaults that block Training and Agent crawlers on ad-supported pages for new domains, new sites on existing accounts, and free-tier accounts that have not changed their settings.
To be fair to Cloudflare: Search crawlers stay allowed by default, so this is not an automatic Googlebot block on 15 September. The risk is narrower and easier to miss. It is that a Training block you set yourself, or inherit from a default, reaches Googlebot through the strictest-rule logic because Googlebot is not purely a search crawler any more. Check the setting rather than assume either outcome.
Cloudflare has also narrowed what “verified” buys a bot. A verified bot is no longer allowed everywhere by default: its access now depends on which category it falls into. That is a meaningful change if your allow rules were written on the old assumption.
What to check today
- Open your Cloudflare dashboard and look at Security, then Bots. Find your AI crawler settings and confirm Search is still allowed.
- If you ever enabled the older block AI bots toggle, that is the setting most likely to catch Googlebot under the new rules.
- A network-level block is stricter than a robots.txt line. Robots.txt is an instruction a crawler chooses to follow. An edge block simply refuses the request, and there is nothing for Googlebot to obey or ignore.
How to allow Googlebot in Cloudflare
Work down this list in order. The first four are settings that block crawlers as a side effect, and they account for most accidental blocks. The fifth is the rule you actually want in place afterwards.
1. Turn off Under Attack Mode
Under Attack Mode makes every visitor pass an interstitial before reaching your site. Googlebot cannot pass it. If the toggle is on, that alone is enough to stop crawling completely, and it is easy to leave on after an incident.
2. Exempt search engines from Browser Integrity Check
Browser Integrity Check inspects headers and browser characteristics, and a crawler that does not look like a browser can fail it. Under Security then Settings, enable the option that stops it applying to search engines.
3. Lower an aggressive Security Level
A Security Level of High challenges traffic on reputation alone. Crawlers hit many URLs quickly from a small pool of addresses, which is exactly the pattern reputation scoring dislikes. Medium is usually the right setting for a site that wants to be indexed.
4. Check country, ASN and rate limiting rules
Googlebot crawls largely from United States addresses on Google’s network, ASN 15169. A country block, a zone lockdown or an ASN rule written for an unrelated reason can take the crawler out. Rate limiting is the sneakiest of the four, because it only fires when Google crawls hard, so the site looks fine every time you test it by hand.
5. Add a Skip rule on Known Bots, not on the user-agent
In a WAF custom rule the field is Known Bots, written in the expression editor as cf.client.bot. Cloudflare only sets it true for bots it has verified by IP, so a Skip rule on that field lets real search crawlers past your other rules. Allowlisting the string “Googlebot” instead would let anyone through, including whatever the rule was written to stop, since the user-agent is free for anyone to send. This tester proves that point every time it runs.
Behind a different vendor, the shape is the same: find the verified-bot or good-bot category and exempt it, rather than matching on a string. The tester above names the vendor it fingerprints so you know which dashboard to open.
How to verify Googlebot for real
Before you allowlist anything, understand how Google says to identify its crawler. Everything else on this page follows from it.
Reverse then forward
Reverse DNS the visiting IP and confirm the hostname ends in googlebot.com, google.com or googleusercontent.com. Then forward-resolve that hostname and confirm it returns the same IP. Both directions must agree, because a reverse record on its own can be forged.
Or match the published ranges
Google publishes its crawler IP ranges as JSON, split by crawler type. For Googlebot the file is common-crawlers.json. Matching against those ranges is the practical approach when you are checking log lines at volume rather than one address.
Never trust the string
The Googlebot user-agent is public and anyone can send it. Scrapers routinely do, specifically to get past bot rules. A user-agent allowlist is a hole, which is why every serious bot-management product verifies by IP instead.
The authoritative check is Search Console
Open Search Console for the property, paste the URL into URL Inspection, and press Test live URL. That fetch is performed by Google, from Google, as Googlebot. It tells you the real status code, whether the page could be fetched and rendered, and what Google actually saw. No external tool can substitute for it, including this one, and any tool that implies otherwise is selling you a guess as a fact.
What an external tester is genuinely good for is speed and breadth. You can check a URL you do not own, check before you have Search Console access, or check a handful of pages in seconds to decide whether a firewall is even worth investigating.
Nobody blocks Googlebot on purpose
Every accidental block has the same story. A managed ruleset updated. A bot-management default changed. A security plugin was installed after an incident. A rate limit was tightened. Under Attack Mode went on during an attack and never went off. None of these are announced, and none of them touch your site code, so there is nothing in your repo to review and nothing in view-source to notice.
That is the gap a one-off test cannot close. It is true about this minute and then it forgets. By the time a traffic drop is large enough to investigate, the pages have usually been gone for weeks.
Watch the outcome, not just the configuration
You cannot monitor every firewall rule your stack might grow. You can monitor the thing that actually matters, which is whether your pages are still in Google. SearchOptimo re-checks your URLs on a schedule, keeps an index history timeline per URL, and alerts you the day a page drops out. Whatever the cause, you find out when it happens rather than at the next audit.
Frequently asked questions
- How do I check if Googlebot is blocked?
- Request the URL twice, changing only the user-agent, and compare the two responses. If the Googlebot user-agent gets a 403 or a challenge page while an ordinary browser gets a 200, something on the path is filtering by user-agent. The tester above does this for you. It is a fast first read, not a final answer: the definitive check is Google Search Console URL Inspection with Test live URL, because that request comes from real Googlebot infrastructure and ours does not.
- Can this tool prove that Googlebot is blocked?
- No, and any tool that says otherwise is overclaiming. We send from a SearchOptimo server IP with a Googlebot user-agent string. Google verifies its own crawler by reverse DNS to googlebot.com or google.com plus its published IP ranges, so a firewall that allows Googlebot by verified IP will refuse us while letting Googlebot through, which reads here as a false positive. The reverse also happens: a rule that blocks on IP reputation rather than user-agent never fires for our request, so a genuine block can pass this test cleanly. What the test can establish is user-agent based blocking, a challenge served to the crawler, a status code difference between the two agents, and materially different content served to Googlebot.
- Is Cloudflare blocking Googlebot on my site?
- It can be, and the usual causes are a small set of settings rather than a deliberate rule. Under Attack Mode challenges every visitor including crawlers. Browser Integrity Check can fail crawlers that do not execute JavaScript. A high Security Level challenges more legitimate traffic. Bot Fight Mode and Super Bot Fight Mode can challenge bots unless verified bots are explicitly allowed. Country or ASN blocking can catch Googlebot, which crawls largely from United States addresses on Google ASN 15169. Check Security then Events in the Cloudflare dashboard, filter for requests whose user-agent contains Googlebot, and read which rule fired.
- How do I allow Googlebot in Cloudflare?
- Use the verified-bot signal rather than a user-agent string. In a WAF custom rule the field is Known Bots, which maps to the expression cf.client.bot, and Cloudflare only sets it true for bots it has verified by IP. A Skip rule matching cf.client.bot lets verified search crawlers past your other rules without opening a hole that anyone can walk through by copying the Googlebot user-agent. Then check Security then Settings and set Browser Integrity Check to not apply to search engines, and confirm Under Attack Mode is off.
- Does blocking AI crawlers also block Googlebot?
- On Cloudflare it now can. Cloudflare has moved from a single block AI bots switch to three behaviours: Search, Agent and Training. From 15 September 2026 it applies the strictest matching rule to crawlers that do more than one thing, and it names Googlebot, Bingbot and Applebot as exactly those crawlers, because each one crawls for both search and AI training. A site that blocks Training therefore also blocks Googlebot. Cloudflare has also changed what verified means: a verified bot is no longer permitted everywhere by default, its access depends on its category. If you turned on the older block AI bots setting and have not revisited it, check it before that date.
- What does "Blocked due to access forbidden (403)" mean in Search Console?
- It means Googlebot reached your server and was refused, so it is almost never a robots.txt problem. A robots.txt block reports separately as Blocked by robots.txt. A 403 points at a web application firewall, a CDN bot rule, a security plugin such as Wordfence, or an origin server config rejecting the request. Start by reproducing it with the tester above, then read your edge logs for the rule that fired.
- How do I verify that a request really came from Googlebot?
- Run a reverse DNS lookup on the visiting IP address and confirm the hostname ends in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it resolves back to the same IP. Both directions have to agree, because the reverse record alone can be forged. Google also publishes its crawler ranges as JSON, including common-crawlers.json for Googlebot, so you can match by IP range instead when you are checking at scale. Never allowlist on the user-agent string alone: anyone can send it, which is exactly what this tester does.
- What is the difference between blocked by robots.txt and blocked by a firewall?
- Robots.txt is a request that well-behaved crawlers choose to honour, and Googlebot honours it. A firewall block is enforced at the network edge, so nothing is being asked and nothing can be ignored. That difference matters when you are diagnosing: a robots.txt block is visible in the file and reversible in one line, while a firewall block leaves no trace in your site code at all and is only visible in edge logs or in the response your server gives the crawler. Use the noindex checker for the robots.txt and directive side, and this tester for the network side.
Find today’s block. Then catch the next one automatically.
Testing by hand catches the rule that is live right now. Monitoring catches the one that has not been written yet. Free forever plan, no credit card, cancel anytime.