What is robots.txt, and what should it contain?

What robots.txt is

robots.txt lives at the root of a host, such as https://example.com/robots.txt. Crawlers fetch it before crawling and follow its rules. The format is standardized in RFC 9309, and major search engines follow it. Each subdomain needs its own file.

What a safe file looks like

For most sites, the right file is almost empty:

User-agent: *
Disallow: /api/
Disallow: /app/

Sitemap: https://example.com/sitemap.xml

That allows every crawler to fetch everything except the API and the logged-in app, and tells them where the sitemap is. If nothing needs excluding, an empty Disallow line or no Disallow lines at all means everything is allowed.

What robots.txt cannot do

• It does not remove pages from Google. A blocked URL can still appear in results, without a description, if other pages link to it. To keep a page out of search, allow crawling and use a noindex tag, or protect it with a login.
• It is not security. Anyone can read the file, and it often advertises the very paths you wanted hidden. Bad bots ignore it.
• It does not apply to other hosts. Rules on www.example.com do not cover app.example.com.

Never block a page in robots.txt and add noindex to it at the same time. If Google cannot crawl the page, it never sees the noindex.

The mistake that removes a whole site

This file blocks every compliant crawler from every page:

User-agent: *
Disallow: /

It is common on staging environments and gets copied to production at launch more often than anyone admits. If organic traffic stops after a release, check robots.txt first.

AI crawlers

Many AI companies publish crawler names and honour robots.txt, for example GPTBot for OpenAI, ClaudeBot for Anthropic and Google-Extended, which controls use of content for Google's AI models without affecting Google Search. Decide deliberately: blocking them keeps content out of some AI training and features, and can also keep your site out of AI answers that cite sources. Each company documents what its user agents do, so check their pages before writing rules.

Common mistakes

• Disallow: / left over from staging.
• Blocking CSS and JavaScript files, which stops Google rendering pages properly.
• A single-page app returning its HTML page at /robots.txt instead of a text file.
• Rules for paths with the wrong case or a missing leading slash.
• Using robots.txt to hide private files that are still publicly reachable.

What a scan can and cannot tell you

WebAuditOS checks that robots.txt exists and returns readable text, and asks a robots parser whether Googlebot may fetch the homepage. A file that blocks the root fails at critical severity. It also checks for pages marked noindex and for a sitemap.

It does not test every path for every crawler, and its parser can differ from Google's on edge cases. If the file blocks the scanner itself, the scan stops before producing a normal report. Search Console's robots.txt report shows how Google read your file.

A file that allows the homepage can still block a whole section. Read your Disallow lines against your real URL structure.

Short answers

Do I need a robots.txt? It is optional, but it is the standard place to point crawlers at your sitemap.

Can robots.txt remove a page from Google? No. Use noindex or remove the page.

Should I block AI crawlers? It is a business decision. Blocking can also keep you out of AI answers that cite sources.

Why does Search Console say indexed though blocked by robots.txt? Google found links to the URL but could not crawl it. Use noindex instead if you want it out.