Short answer
Robots.txt is a plain-text file at the root of your website that asks crawlers which URLs they may fetch. It controls crawling, not indexing, and it is not security: a blocked page can still appear in search, and the file is public. For most small businesses a short file that allows everything, blocks only admin areas and lists the sitemap is the safe default.
What is robots.txt, and what does it do?
Robots.txt is a plain-text file at the root of your website, for example https://www.example.com/robots.txt. Before a well-behaved crawler fetches your pages, it reads this file for rules about which addresses it may request. Each rule names a crawler and a path. The file applies only to the host it sits on, so a subdomain such as shop.example.com needs its own. Googlebot and Bingbot follow it, and nothing forces a bad bot to.
What can robots.txt not do?
It controls crawling, not indexing. If you block a page, Google cannot read it, but it can still list the address in results, without a description, when other sites link to it. To keep a page out of search, leave it crawlable and use a noindex tag, or put it behind a login.
What does robots.txt syntax look like?
| Line | What it does |
|---|---|
User-agent: * | Starts a group of rules for every crawler. Use a name such as Googlebot for one crawler. |
Disallow: /cart/ | Asks crawlers not to fetch addresses that begin with /cart/. |
Allow: /wp-admin/admin-ajax.php | Makes an exception inside a blocked folder. The most specific matching rule wins. |
Disallow: with nothing after it | Blocks nothing. |
Disallow: / | Blocks the entire site. Rarely what you want. |
Sitemap: https://www.example.com/sitemap.xml | Tells crawlers where your sitemap is. Use the full address. |
Paths are case-sensitive and match from the start, so Disallow: /blog also blocks /blog-news/. Google also understands * as a wildcard and $ as the end of an address.
What is a safe robots.txt example for a small business?
Allow everything that matters, block only areas with no search value, and point to your sitemap. For a WordPress site that can be these four lines, with your own address in the last one:
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.phpSitemap: https://www.example.com/sitemap.xml
Add Disallow: lines for cart, checkout and internal search pages if you have them. Do not block your CSS, JavaScript or image folders, because Google needs them to see pages as visitors do. If you have nothing to block, a file with only the Sitemap: line, or no file at all, is fine.
How do you test robots.txt in Search Console?
- Open
/robots.txtin your browser and read it. It should load as plain text, not an error page. - In Search Console, open the robots.txt report under Settings. It shows the file Google last fetched, when, and any errors or warnings.
- Run the URL Inspection tool on an important page. It says whether Google may crawl it or robots.txt blocks it.
- Re-test after every edit. Google generally caches the file for up to a day, so changes take a little while to show.
What about AI crawlers in robots.txt?
Robots.txt is also where you allow or block AI crawlers, each with its own name. For example, User-agent: GPTBot followed by Disallow: / blocks OpenAI's training crawler. The names to know, as of late 2026, are:
- OpenAI:
GPTBot(training),OAI-SearchBot(ChatGPT search),ChatGPT-User(fetches a page when a person asks) - Anthropic:
ClaudeBot,Claude-SearchBot,Claude-User. Perplexity:PerplexityBot,Perplexity-User - Google:
Google-Extended(Gemini, no effect on Search ranking). Apple:Applebot-Extended. Common Crawl:CCBot
Blocking training bots does not remove you from Google Search, and OpenAI says robots.txt rules may not apply to ChatGPT-User. Check each vendor's page, because names change. The choice is a business decision: see should you allow or block GPTBot and other AI crawlers.
What are the classic robots.txt disasters?
- A leftover
Disallow: /from a staging site that went live with its file. The whole site stops being crawled, and it can go unnoticed for weeks. It is a classic cause of a website not showing on Google. - A CMS setting that discourages search engines, left on after launch.
- Blocking CSS, JavaScript or image folders, so Google cannot render pages properly.
- Using the file to remove pages from Google. It cannot. Use noindex or a removal request.
Arcline reads your robots.txt on every scan and flags rules that block the crawlers that answer people's questions, such as ChatGPT, Perplexity and Claude. Editing robots.txt is a sensitive change, so Arcline prepares the corrected file and applies it only after your approval. See how the checks work.