We Read 30 Days of AI Crawler Logs on Our Own Website

Every week somebody on LinkedIn tells small business owners what the AI crawlers want from their website. Add this file. Restructure that page. Most of it is confident and none of it comes with logs.

We have logs. Our own website runs on a server we control, so every request that hits it gets written down: who asked, what they asked for, when. We pulled 30 days of those records, August 13 through September 12, and looked at what the crawlers from OpenAI, Anthropic, Perplexity, and Apple actually did.

Here's what they did.

They showed up constantly

Our main site took about 20,800 requests in those 30 days. Not visits, requests: every page, image, and file counts as one.

The crawler that identifies itself as ClaudeBot, Anthropic's, made 722 of them. Google's own crawler made 583. Bing made 493. OpenAI's three crawlers together made 364. Apple's made 160. Perplexity's made 20.

So the AI companies are crawling a small marketing shop in a town of 1,700 people about as hard as Google is. Whatever you think of that, it's happening, and it's happening to your site too.

They read two files over and over

Here's the part that surprised us. ClaudeBot's 722 requests were almost entirely two files: robots.txt, 332 times, and sitemap.xml, 332 times. It checked the rules and the map roughly every two hours, and only occasionally opened an actual page.

Google did the same thing at a slower pace: 121 robots.txt pulls, 46 sitemap pulls. Bing, 102 and 112. OpenAI's search crawler pulled robots.txt 131 times.

Those two files are the ones a lot of small business sites get wrong or leave out. In the Iowa study we published earlier this month, 4 of the 57 sites had a robots.txt problem (three had no file at all, one blocked the AI crawlers outright), and 35 of them had a missing or weak meta description, which is the next thing a crawler reads after the map. That study measured sites we don't own. This measured ours, and the bots told us the same thing: the rules and the map are what they come back for.

The file none of them asked for

If you've read anything about "AI readiness" in the last year, you've heard of llms.txt. It's a plain-text summary of your business that sits at the root of your site so an AI tool can read it instead of guessing from your pages. We have one. Our free website checker at mainstreetaudit.com has told people for a month that they should have one too.

In 30 days, across our main site, our blog, and our tools site, the number of times any AI crawler requested that file was zero.

Not a rounding error. Zero. The 66 requests it did get came from our own scanner, a couple of SEO tools, and a handful of people in web browsers, probably curious after reading about it.

We're not the only ones seeing this. An SEO named Peter Rota ran the same test on a client site and got two hits in a month. The file isn't harmful. It costs nothing. It's just not something the crawlers fetch today, and anyone who tells you it's urgent is guessing.

So we changed our own tool

Our checker used to flag a missing llms.txt as a warning, the same color as a missing title tag. After reading our own logs, that's not honest. We've downgraded it to an information note. It still offers to build the file for you in one click, because why not, but it now says plainly: the crawlers read robots.txt and your sitemap every day and haven't asked for this once. Fix the first two before you think about the third.

If a tool we built was overstating something, we'd rather say so than let the warning sit there looking important.

Two more things the logs showed

Our blog gets its own share. Apple's crawler made 382 requests there and loaded the pages the way a browser would, CSS and scripts included. OpenAI's search crawler and Perplexity read individual posts, one at a time, the way a person would. The guide to rural SEO, the piece about showing up in your community, the one about small business owners showing their face. Those are the ones that got read. Plain answers to plain questions.

And a warning for anyone reading their own logs: nearly 500 of the 621 requests claiming to be OpenAI's search crawler came from addresses that aren't OpenAI's. Those were vulnerability scanners wearing a costume, probing for leaked configuration files. Every major crawler except Anthropic's publishes the addresses it uses. We checked every request against those lists, and checked ClaudeBot's addresses against the public network registry (they sit in a block registered to Anthropic), before counting anything. If you ever pull your own logs, do the same, or the numbers will lie to you.

What this means if you own a small business

You don't need to do anything exotic to be readable by AI tools. You need the same two things you needed for Google in 2015: a robots.txt file that lets crawlers in and names your sitemap, and a sitemap that lists your real pages. Then pages that answer real questions in plain language.

That's it. That's what the bots came for 30 days in a row.

You can see whether your own site has those two files in about a minute with the free checker at mainstreetaudit.com. If it flags them, the fixes are small. If you'd rather hand it to somebody, that's what we do. Call or text (515) 422-6208.