Cloudflare could hide your site from Google Sept 15

There is a setting sitting in front of roughly a quarter of the internet that can remove your business from Google, and the people most likely to have flipped it are the ones who thought they were doing something responsible.

On September 15, Cloudflare changes how it treats crawlers. If somebody at some point checked the box that blocks AI training on your site, that box is about to start blocking Googlebot too. Not the AI answer. Googlebot. The crawler your rankings are made of.

I want to be careful here, because this story has been covered badly in a few places and the sloppy version will scare people who are not actually at risk. So let me walk it exactly as Cloudflare wrote it.

What Cloudflare actually changed

Since July 1, every Cloudflare customer, including everyone on the free plan, can sort crawlers into three buckets instead of one blunt on and off switch. Cloudflare's own definitions:

• Search: "any behavior that collects or indexes your content, so it can answer questions about it later."
• Agent: "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now." This is the bot that fetches your page mid conversation when somebody asks ChatGPT about you.
• Training: "a crawler taking your content to train or fine-tune a model."

That is a genuinely good taxonomy. For two years the only choice was allow everything or block AI, and blocking AI meant blocking the thing that recommends you. Splitting Agent away from Training is the fix a lot of us have been asking for.

Then comes the part that matters.

The trap is the mixed-purpose crawler

Googlebot does two jobs with one user agent. It indexes your site for search, and it feeds Google's AI products. Cloudflare calls that a multi-purpose crawler, and on September 15 it starts judging those crawlers by everything they do rather than the friendliest thing they do.

From Cloudflare's announcement, word for word:

"Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)."

Read that twice. It names Googlebot. It names Applebot, which now crawls from 4,656 newly added IP addresses that Apple quietly published, per Barry Schwartz at Search Engine Roundtable on August 17. And it names the legacy toggle, which is the one people actually clicked, because for about two years "Block AI bots" was a single friendly looking switch that a lot of web developers turned on without a second thought.

This is not theoretical. Search Engine Journal reported on August 4 that a site owner in r/TechSEO had Googlebot and Bingbot both returning 403 errors on their sitemap after enabling AI training blocking. John Mueller from Google showed up in the thread asking to take it to DMs. When Mueller wants to move a crawl issue off a public forum, something real is going on.

Glenn Gabe flagged the underlying fight on X with the only two words that matter here: be careful.

Does this even apply to you?

At this point a lot of owners reading this are thinking "I have no idea what Cloudflare is, so I am probably fine." Half of you are right. Let me sort it out, because I would rather you skip a task than do a panicked thing to a site you do not understand.

Cloudflare sits in front of 24.2% of all websites as of late July, according to W3Techs, and 84.1% of the sites running any identifiable reverse proxy. So the odds are not small. But whether the risk lands on you depends on how your site was built.

Probably not affected: Wix, Squarespace, and Shopify sites. Those platforms manage their own infrastructure and you were never handed a Cloudflare dashboard, so there is no toggle for anyone to have flipped.

Worth checking: self hosted WordPress, anything a freelance developer set up for you, anything on a host that resells Cloudflare, and any site where somebody proudly told you they "blocked the AI scrapers" at some point in the last two years. That last group is the one I actually worry about. The person who did it meant well.

Definitely check if you got an email from Cloudflare about default changes and archived it. They have said they will keep notifying customers as September 15 approaches.

The three minute check

You need one thing: whoever has the login. If that is you, go to your Cloudflare dashboard, pick the domain, and open Security settings. That is the same screen Cloudflare points to in its own opt-out instructions. Look for the AI crawler controls and confirm that Search is allowed. If Training is blocked and you want it blocked, fine, but you need to mark the opt-out so Cloudflare does not extend that block to crawlers that also do Search.

If you do not have the login, send your developer this, verbatim:

"Before September 15, please confirm in Cloudflare Security settings whether we are blocking AI Training, and whether we have opted out of the new multi-purpose crawler defaults. I want Search allowed for Googlebot no matter what we do with Training. Please send me a screenshot of the setting."

Ask for the screenshot. Not a "yep we're good." I have been told "yep we're good" about robots files that were blocking entire sites.

Then verify from Google's side, which is the part most people skip. Open Search Console, run the URL Inspection tool on your homepage, and hit Test Live URL. If Googlebot cannot fetch the page, you will see it there in about twenty seconds. That single test is worth more than any assurance you will get from a human being.

My actual opinion

I think Cloudflare is doing the right thing and it is still going to hurt a bunch of small businesses.

The right thing: they are being honest that Googlebot is not a pure search crawler anymore. It has not been for a while. Every "block AI training" tool on the market has been quietly lying to you about that, letting you feel protected while Google took the content anyway. Cloudflare enforcing the most restrictive rule is the only intellectually consistent position, and they gave everyone ten weeks and an opt-out.

The part that will hurt: the businesses that flipped that switch are, by definition, the ones who do not read Cloudflare changelogs. They clicked a checkbox in 2024 because a blog post told them AI was stealing their content. Nobody is going to tell them their Charleston plumbing site fell out of Google in late September. They will just notice the phone stopped ringing, and by the time they connect it to anything they will have lost a month of storm season work.

This is the same shape as the Google AI Overviews opt-out I told most of you to leave alone back in June. The instinct to pull your business out of AI feels protective. For a local service business it is almost always the wrong trade, and here it is worse, because this time the cost is not an AI mention. It is your organic rankings.

It also lands in the same place as what I found testing whether schema markup moves Charleston local SEO: the technical work that actually pays is boring plumbing, not the shiny thing. Nobody gets excited about verifying that Google can reach your homepage. It is still the highest leverage twenty seconds in your entire marketing operation, and it is exactly the sort of thing we build into every Charleston web design project before anyone talks about content.

Do this before Labor Day

1. Find out if your site is behind Cloudflare. Ask your developer, or check whether your DNS points at Cloudflare nameservers.
2. If it is, open Security settings and confirm Search is allowed and you have handled the opt-out.
3. Run URL Inspection on your homepage in Search Console and click Test Live URL. Do this whether or not you use Cloudflare, because a blocked crawler can come from six other places.
4. Put a reminder on September 16 to run that same live test again. One minute, and it catches this whole category of problem.

If you run a Charleston roofing company heading into the back half of hurricane season, that fourth item is not optional. Losing search visibility in September is a different kind of expensive than losing it in February.