Which AI crawlers actually visit small business websites?

What we measured

Every page we host is served through our own infrastructure, so a crawler fetch is a fact in our logs rather than something inferred. We record the declared user-agent, the page path, and the day, aggregated per site. No third-party tool sits in the middle.

The corpus for this report is 599 newly built small business websites — service businesses, restaurants, clinics, trades, mostly in the north-eastern United States. Every one of them was live and publicly reachable for at least seven days during the window, was listed in a sitemap, and — this is the important part — had no inbound links from anywhere else on the web.

That makes them a natural experiment in something usually impossible to isolate: what finds a website purely because it exists, with no reputation, no links, and nobody talking about it. Every new website starts in exactly this position.

Across the whole estate we recorded 5,172 fetches from 10 distinct crawler products between 30 July and 19 August 2026.

Finding 1 — Three crawlers do nearly all the visiting

Of the 599 unlinked sites, here is how many each crawler fetched at least once during the three weeks:

Fetch volumes tell a slightly different story from reach. Bing was almost as broad as Meta but far lighter — 181 fetches across 176 sites, roughly one look each. Meta and Claude went back repeatedly.

Finding 2 — Meta runs the busiest crawler nobody discusses

meta-externalagent generated 1,848 of the 2,716 fetches against our unlinked corpus — more than Claude, Bing, and Google combined. On 31 July, the first full day of measurement, it fetched 172 of them in a single day — 1,573 of its fetches in 24 hours.

It is striking how little attention this gets. Search for advice on AI crawlers and you will find a great deal about GPTBot and almost nothing about the crawler that, in our data, is doing the most work by a factor of nearly three.

Finding 3 — A sitemap is not discovery

Every one of the 599 sites was listed in a sitemap for the entire window. Google fetched three of them.

This is worth sitting with, because "submit a sitemap" is standard advice for a new site. A sitemap tells a crawler that a URL exists. It does not give it a reason to care. In our data, being listed and being crawled turned out to be almost unrelated for pages with no links pointing at them.

Finding 4 — ChatGPT and Perplexity do not sweep

Zero fetches across 599 publicly reachable, sitemap-listed sites, over three weeks.

Both crawlers were highly active elsewhere on our infrastructure during exactly the same window — on the sites in our estate that were live on their own domains and linked from the wider web. We are deliberately not quoting a percentage for that group: it is a handful of sites, far too few to generalise from, and a number would imply a precision we do not have.

What the contrast supports is a directional claim, and it matches how these systems are documented to work: ChatGPT and Perplexity are discovery-driven rather than sweep-driven. They follow links and existing indexes. Publishing a page is not an event they observe.

Finding 5 — Most of ChatGPT's traffic is not training data

OpenAI crawls under three names, and they do different jobs. Across our whole estate:

53% of OpenAI's fetches came from the two agents that are not training. That matters because blocking GPTBot is widely recommended as a privacy measure and just as widely misread as cutting yourself off from ChatGPT. It does not: it declines training use while leaving search indexing and live retrieval untouched.

The 145 ChatGPT-User fetches are the most interesting number in this report. Each one is ChatGPT going to read a page because a person in a conversation asked something that required it. That is not a crawler building an index — that is a customer, mid-question.

Finding 6 — Crawlers read past the homepage

Across the estate, 4,493 fetches landed on pages other than the homepage, against 679 on homepages. Crawlers went looking for services pages, location pages, and articles.

The common instinct to put all the effort into the homepage and treat everything else as filler is backwards. The interior pages are what got read.

Finding 7 — A quarter of sites were never visited at all

164 of the 599 sites — 27.4% — received no fetch from any crawler in the entire window, despite being live, reachable, and listed.

There is no penalty at work here and nothing was misconfigured. They were simply pages nothing had a reason to look at. For a new business website, that is the default state, and it persists until something changes it.

What we would do with this

1. Stop treating publication as discovery. A new site with no links is invisible to the assistants most people care about. Getting one external mention matters more than another week of on-page work.
2. Do not panic about GPTBot. Blocking it costs no citations. Check that OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, and Googlebot are allowed instead.
3. Write the interior pages properly. They are what gets fetched.
4. Know that Meta and Claude will find you early. They sweep. If your pages are thin or unreadable when they arrive, that is what they take away.

Limits of this data

We would rather state these plainly than have them found for us.

• 21 days is a short window. Crawl behaviour changes; treat these as a snapshot, not a constant.
• One platform, one edge. Everything here is what reached our infrastructure. A different host with different bot protection would see a different picture.
• One market. Predominantly north-eastern US small businesses. We would not extend it to other geographies or to e-commerce.
• The corpus is newly built sites. Some of these pages resemble the businesses' existing websites, and a crawler that detects near-duplicate content may deprioritise it. This could depress the numbers, and we cannot separate that effect from simple lack of interest.
• The comparison group is tiny. As above — the linked-site contrast is directional, not measured.
• Crawlers can lie. We record declared user-agents and do not verify by IP, so a spoofed agent would be counted. For the major operators this is a minor concern; we mention it for completeness.

No individual business, domain, or site is identified anywhere in this report, and nothing here is derived from the content of anyone's website — only from counts of who fetched what.

Common questions

Which AI crawlers visit small business websites most?

In our data, Meta's meta-externalagent by a wide margin (1,848 fetches), then Claude (684) and Bing (181). ChatGPT and Perplexity did not fetch unlinked small business sites at all during the window.

Why doesn't ChatGPT crawl my new website?

Because it does not sweep for new sites. It follows links and existing search indexes. Until something else on the web points at your site, it has no path to it — publishing alone is not an event it observes.

Does blocking GPTBot stop ChatGPT from citing me?

No. In our data GPTBot accounted for 47% of OpenAI's fetches; the other 53% came from ChatGPT-User and OAI-SearchBot, which handle live retrieval and search indexing. Blocking GPTBot declines training use only.

How long does it take for AI crawlers to find a new site?

For sweeping crawlers like Meta, Claude, and Bing, days. For ChatGPT, Perplexity, and Google, our data suggests the honest answer is that it does not happen on a timetable at all — it happens when something links to you.

Is being in a sitemap enough to get crawled?

Apparently not. All 599 sites were sitemap-listed for the full window and Google fetched three of them. A sitemap declares that a URL exists; it does not create a reason to visit.