AI crawlers on your classifieds site: what actually decides which ones to block
The bots in your server log that aren't Google
Pull up the raw access log on a classifieds directory for even one day and the list of visitors stops looking like a list of people. Alongside Googlebot and the usual browsers there's GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, and half a dozen names that mean nothing until you look them up. They hit every listing page, including the ones that changed an hour ago, and they do it on a schedule no human visitor follows.
An adult classifieds directory is a better target for this kind of crawling than most sites, for reasons that have nothing to do with the content being adult. Listings change constantly, the city and category structure multiplies the number of distinct pages, and each one carries photos and text that a crawler can harvest in one pass. The same features that make the site useful to a visitor make it efficient to scrape.
Robots.txt is the file that is supposed to settle this: a plain-text list of rules telling each named crawler what it may fetch. It sits at the root of the domain, any crawler can read it, and the ones built by companies that say they respect it, OpenAI, Anthropic, Google, generally do. But the file is a request, not a lock. Nothing in the protocol stops a crawler from fetching a disallowed page anyway, and nothing on the website's side enforces the rule unless the operator adds something else on top of it.
That gap matters more than it used to, because the crawlers showing up in that log aren't one thing. Some want to train a model on the page. Some want to answer a question about it right now. Some are a person's own request routed through an assistant. Treating all three the same, by blocking everything or blocking nothing, throws away a choice an operator can actually make.
Training, answering, and "a user asked" are three different jobs wearing one name
OpenAI runs three separate crawlers, and conflating them is the most common mistake in a hastily written robots.txt. GPTBot collects pages that may be used to train future models, and OpenAI's own documentation says disallowing it keeps a site's content out of that training data. OAI-SearchBot is unrelated to training: it's what lets a page show up as a cited source in ChatGPT's search feature, and it is controlled separately from GPTBot, so a site can block training while staying eligible to be cited. ChatGPT-User is neither: it fires when an actual person asks ChatGPT a direct question about a page, and OpenAI states plainly that robots.txt rules may not apply to it, because the fetch is a response to a specific human request rather than automated crawling.
Anthropic draws the same three-way line under different names. ClaudeBot crawls for training, and Anthropic's support documentation says its bots honor robots.txt directives on that front. Claude-User and Claude-SearchBot cover the live-question and search-citation cases respectively, each with its own opt-out.
Google's version is narrower but easy to get backwards. Google-Extended isn't a crawler at all; it's a token added to robots.txt that controls whether content Googlebot already fetched can be used to train Gemini or ground its answers. Disallowing it does not touch a site's ranking or inclusion in ordinary Google Search, because Googlebot keeps crawling exactly as before. What changes a classifieds directory's odds of showing up when someone's question gets answered inside Google's AI Overviews instead of in a list of blue links is a separate setting, not Google-Extended: Overviews draw on the regular Googlebot crawl, so blocking the Extended token opts a site out of Gemini training without opting it out of Overviews. The only lever that touches Overviews and ordinary search snippets together is the nosnippet family of tags, which is a bigger trade-off than most operators realize before they flip it.
The practical upshot is that an operator can write a robots.txt that disallows every named training crawler while leaving the search and citation crawlers untouched, getting the one outcome without the other. Nobody sets that distinction up by accident; it has to be written in by hand, bot by bot.
What Cloudflare's new policy adds, and what it still can't do
Robots.txt was built to answer one question: which pages can you fetch. It was never built to answer a second one: what are you allowed to do with the page once you have it. Cloudflare's Content Signals Policy, introduced in September 2025, adds exactly that second layer. A site can now add a Content-Signal line under any user-agent group with three values, search, ai-input, and ai-train, each set to yes or no. Leaving one out means the operator hasn't stated a preference for that use, not that the answer is no.
This is useful specifically because it lets an operator say, in one place, "index this for search, but don't feed it into a model and don't train on it," which is a distinction the old robots.txt syntax had no vocabulary for. Sites on Cloudflare's managed robots.txt service got search=yes, ai-train=no added automatically for the roughly 3.8 million domains that use it; anyone else can write the lines by hand or generate them at contentsignals.org.
The honest limitation is in Cloudflare's own description of the policy: these are declared preferences, not a technical barrier, and a crawler that chooses to ignore them faces nothing stopping it beyond whatever the site has layered on separately, like a firewall rule or bot-management product.
That limitation stopped being theoretical a month before the policy even shipped. In August 2025, Cloudflare published a report accusing Perplexity of running what it called stealth crawlers: bots that didn't fetch robots.txt before crawling disallowed test domains, and that switched to a generic Chrome-style browser signature once the declared PerplexityBot was blocked, routed through IP ranges Perplexity hadn't published. Cloudflare removed Perplexity from its verified-bot program over the findings. Perplexity called the report a publicity stunt and disputed the attribution, but didn't rebut the specific user-agent and IP evidence point by point. Whichever side was more right, the episode is the clearest demonstration available that a declared rule, in robots.txt or in a Content-Signal line, only binds a crawler that has decided to be bound by it.
The two costs an operator actually carries
The first cost is the one that shows up on an invoice. A classifieds directory running on a budget VPS or a shared-hosting plan, which is where most operators in this category actually run, typically has a monthly data-transfer allowance rather than an unlimited pipe, and a crawler counts against that allowance exactly the same as a paying visitor does. A training crawler re-fetching every listing and photo on a multi-city directory on a recurring schedule adds load that has nothing to do with the people the site exists to serve, and the same infrastructure fragility that lets a host or CDN end a classifieds business with a termination notice also means an account that trips a usage-based suspension over bot traffic is exposed to exactly that kind of abrupt shutdown. Imperva, a bot-management vendor, reported in its 2026 Bad Bot Report that automated traffic made up just over half of all measured web traffic, with roughly 40 percent of that total classified as malicious; the figure comes from one vendor with a commercial stake in the problem, but the direction, that a meaningful share of a site's bandwidth bill is not human traffic at all, lines up with what shows up in any raw access log.
The second cost doesn't show up on an invoice at all. A deletion request against the directory's own database is something an operator can actually execute: find the record, remove it, confirm it's gone. A photo and a bio that a training crawler has already collected into a dataset meant to teach a model general patterns has no equivalent request to send. There is no support address for a training corpus, and no confirmation that a specific advertiser's image was or wasn't retained once it left the server. That is a different, harder-edged version of the control problem that already applies to what a classifieds site leaks about an advertiser through the ordinary metadata attached to a photo: once content is out, the operator's ability to account for where it went drops sharply, and an advertiser who asks "where does my listing's photo end up" deserves an honest answer about that limit rather than a reassurance the operator can't back up.
What to actually do about it
Start by reading the current robots.txt, not assuming what it says. A surprising number of classifieds sites built from a template either have no file at all, which leaves every crawler to decide its own limits, or carry a single blanket "User-agent: *, Disallow: /" left over from a staging environment, which blocks Googlebot along with everything else and quietly zeroes out the site's entire organic search channel without any error message to notice it by.
Decide the training question first, because it's the one with a real answer. Write explicit disallow rules for GPTBot, ClaudeBot, Google-Extended, Bytespider, and CCBot at minimum, since those cover the major foundation-model training crawlers with public, documented user-agent strings. This is a decision an operator can make deliberately rather than by default.
Decide the search and citation question separately, and actually decide it rather than copying whatever rule handled training. OAI-SearchBot, Claude-SearchBot, and PerplexityBot determine whether the directory can be cited when someone asks an AI assistant a question that a listing could answer; blocking them trades away that visibility, allowing them keeps it, and there's no reason the answer has to match the one given to GPTBot and ClaudeBot on the line above it.
If the site runs on Cloudflare, check whether the Content-Signal lines were added automatically and whether the defaults match the operator's actual preference, since the automatic setting is sensible but generic, not tailored to one site. If it doesn't, the lines can be written by hand in a few minutes or generated at contentsignals.org and pasted in.
Finally, treat the robots.txt and Content-Signal rules as the easy ninety percent, and accept that the remaining crawlers, the ones willing to do what Cloudflare accused Perplexity of doing, aren't going to be stopped by a text file no matter how carefully it's written. Catching those requires watching for a declared bot's name arriving from an IP address outside its published range, which a CDN's bot-management feature can flag automatically if the budget allows for one, or which shows up by hand in the access log for anyone checking it periodically. Either way, the goal isn't eliminating every unwanted crawler, which isn't achievable; it's making sure the ones who will actually listen are told, clearly, which of the three jobs they're allowed to do.
