Robots.txt and XML Sitemaps: The Complete Guide to Crawl Control

Before a search engine can rank your pages, it has to find and crawl them. Two files sit at the center of that process: robots.txt, which tells crawlers where they're allowed to go, and your XML sitemap, which tells them where your important pages actually are. Used well, they make indexing faster and more efficient. Used badly, they can hide your entire site from Google without you ever knowing.

What robots.txt Actually Does

robots.txt is a plain-text file at the root of your domain (e.g., https://example.com/robots.txt) that gives crawlers directives about which paths they may or may not request. It's a voluntary protocol — well-behaved crawlers like Googlebot respect it, but it is not a security mechanism. Anyone can still request a disallowed URL directly, and the file itself is publicly readable, so never use it to "hide" sensitive paths.

Basic syntax:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public-status/

Sitemap: https://example.com/sitemap.xml

• User-agent targets a specific crawler (Googlebot, Bingbot) or all of them (*)
• Disallow blocks a path prefix from being crawled
• Allow carves out an exception within a disallowed path
• Sitemap points crawlers to your sitemap (or sitemap index) — you can list more than one

Common robots.txt Mistakes

• Blocking CSS/JS: A blanket Disallow: /assets/ can prevent Google from rendering your page correctly, which hurts how it evaluates content and mobile-friendliness.
• Disallow vs. noindex confusion: Disallowing a URL stops crawling, but if the page is already indexed or gets linked from elsewhere, it can still appear in search results with no description ("blocked by robots.txt"). To actually remove a page from the index, use a noindex meta tag or header instead — and don't disallow the same page in robots.txt, since a crawler that can't fetch the page can't see the noindex tag either.
• Wildcard typos: Disallow: / with no exceptions blocks the entire site — a surprisingly common accident when moving a staging robots.txt to production.
• Case sensitivity: Paths are case-sensitive. /Products/ and /products/ are different rules.

What an XML Sitemap Does

A sitemap is the opposite signal: instead of restricting crawlers, it proactively lists the URLs you want indexed, optionally with metadata like last-modified date, change frequency, and priority. It doesn't guarantee indexing, but it dramatically improves discovery — especially for large sites, new pages, or pages with few internal links pointing to them.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-07-01</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
</urlset>

Sitemap Indexes for Large Sites

A single sitemap file is capped at 50,000 URLs and 50MB uncompressed. Sites larger than that need a sitemap index — a file that itself lists multiple sitemap files, each under the limit. This is how large, programmatically generated sites (news archives, e-commerce catalogs, or a site indexing millions of scanned domains) stay crawlable without hitting the per-file ceiling.

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap/1.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap/2.xml</loc>
  </sitemap>
</sitemapindex>

Best Practices

• Only include canonical, indexable URLs in your sitemap — leave out redirects, non-canonical duplicates, and pages marked noindex
• Keep lastmod accurate; a sitemap that always claims "today" for every URL trains crawlers to ignore the field
• Reference your sitemap in robots.txt and submit it directly in Google Search Console and Bing Webmaster Tools
• Regenerate sitemaps automatically as content changes rather than maintaining them by hand

How SiteGraph Checks This

Every SiteGraph scan fetches and parses a site's robots.txt and sitemap.xml, checking whether they exist, whether the sitemap is properly referenced, and whether any rules look like they're accidentally blocking important content. This feeds directly into the SEO score on a site's report page.