Someone in Sellersville needs a lawyer to settle a parent's estate. They don't open Google. They ask the AI assistant on their phone, the way they'd ask a neighbor, and it names two or three firms.
To write that answer, the assistant has to read those firms' websites. If your site says it can't, you're not one of the names. You didn't rank low. You weren't considered. And nothing on your end tells you it happened.
A small file on your website decides this. Most owners have never seen it. Here's how to check yours in thirty seconds, and what to do if it says no.
What the robots.txt file does
Almost every website has a text file called robots.txt. It sits at the top level of your domain, and it tells robots which parts of the site they may read. It's a sign on the door, not a lock, and the big search and AI companies say they follow it.
The file uses three words. User-agent names a robot. Allow means come in. Disallow means stay out. Each Allow or Disallow applies to the visitor named on the line above it.
How to check yours in thirty seconds
Type your web address into a browser with /robots.txt on the end, like yourbusiness.com/robots.txt. What comes back is what every robot has been reading.
If the page is empty or doesn't exist, you're fine. Nothing has been blocked. If there's text, look for the word Disallow, then look at which name sits above it. If you want to see a file that plainly says yes, ours is at corcoranpr.com/robots.txt. It names every robot we want in, one at a time.
The names that matter
Three groups of names show up in these files. They do different jobs, and the difference matters.
The three that put you in AI answers
OAI-SearchBot is how ChatGPT finds websites. Claude-SearchBot is how Claude finds them. PerplexityBot is how Perplexity finds them. If one of these is under a Disallow, that assistant can't read your site, and it answers questions about your trade from whatever else it can find about you. For a small business, that's often nothing.
The two that put you in regular search
Googlebot and Bingbot are ordinary search. A site that blocks them isn't in search results at all. That's a bigger problem than the AI one, and it looks the same from the outside: the phone stops ringing and nobody knows why. Check for these two in the same thirty seconds.
The ones that only train
GPTBot and ClaudeBot collect pages that may be used to train OpenAI's and Anthropic's models. Google-Extended and Applebot-Extended are switches that tell Google and Apple whether your pages may be used for their AI. None of the four is how an assistant finds you when a customer asks a question. Blocking them keeps your writing out of training. It doesn't take you out of answers.
How the block got there
Most of the time, nobody chose to block AI. Wix, Squarespace, and WordPress write this file for you from your settings, and whatever a setting or a plugin put there is what you have. Squarespace, for one, has a single box that blocks AI crawlers. A site that was built on a staging address often keeps the block that hid it while it was unfinished. And one article about AI companies copying other people's work is enough to make a helpful person paste in a block list and move on.
Those block lists usually name every bot at once, the training ones and the answer ones together. That's how a business ends up invisible to ChatGPT without anyone deciding it should be.
None of that was a decision about your business. The question now is whether what the file says is what you'd choose.
You can refuse training and still be in answers
Some owners don't want their writing used to train AI models. That's a fair choice, and you can make it without disappearing. Block GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended if you want. Leave OAI-SearchBot, Claude-SearchBot, and PerplexityBot alone. You keep your words out of training and stay readable when a customer asks.
The other switch, in Search Console
There's one more place a block can hide. Search Console, Google's free dashboard for site owners, has a setting that can take your site out of Google's AI features. It's not in any file on your website. Anyone with access to that dashboard can flip it. So part of knowing what your site says is knowing who has that access, and checking that the setting reads Include.
Why it costs more in a small town
Ask an assistant for an estate lawyer in Philadelphia and it's cutting a long list down to three. Ask for one in Quakertown, Perkasie, or Coopersburg and there may only be a handful of real choices to begin with. Being unreadable in a small town means being left off a short list.
The firm one town over fills your spot. A client in Quakertown will drive to Perkasie, so the assistant names the Perkasie firm it can read. The phone call, the form fill, and the signed engagement land there. Neither firm ever knows.
Where to start
Check your file today. Type your address with /robots.txt on the end and read it. If OAI-SearchBot, Claude-SearchBot, or PerplexityBot sits under a Disallow, ask whoever manages your website to take it out. It's a one-line change.
Then remember what the file can and can't do. Letting an assistant read your site doesn't get you named in its answer. It only makes that possible. What gets you named is a claimed Google listing, reviews with real words in them, being listed where the assistant already looks, and pages that answer questions in plain sentences. That's the longer job, and it's where the phone calls, form fills, and booked jobs come from.
Permission is the part you can settle today, by yourself, in less time than it took to read this. If you'd rather we check it, along with everything else your site is telling search engines and AI, our free site audit will tell you in plain English.