Sitemap and robots.txt: Mistakes That Cut Your Traffic

Robots.txt is twenty lines long and sitemap.xml is generated automatically — what could possibly go wrong. In practice these are the two most frequent sources of disaster: one extra line can remove an entire site from search, and it happens regularly after releases.

What robots.txt actually does

It controls crawling, not indexing. A page blocked in robots can still end up in search if external links point to it — the crawler simply will not see its content and will show an empty snippet. To keep a page out of the index for certain you need a noindex meta tag, and the page itself has to stay open to crawling, otherwise the crawler will never read that tag. This logic is described in detail in Google's robots.txt documentation.

Common mistakes in robots.txt

• Disallow: / in production — left over from the staging server, it removes the site from search within days
• Blocking /wp-content/ or /assets/ entirely: the crawler cannot load CSS and JS and sees a broken page
• Blocking pages that need to leave the index instead of using noindex — they just stay in the results
• No Sitemap directive — the crawler takes longer to find new pages
• Different rules for User-agent: Yandex and Googlebot that nobody remembers a year later
• The file returns 404 or 500 — some crawlers treat that as a ban on crawling the whole site

What belongs in sitemap.xml

Only canonical URLs that return 200 and are open to indexing. No redirects, no noindex pages, no duplicates with parameters. The lastmod date has to be real: if it updates on every page every time the site is rebuilt, the crawler stops believing it. The limits are 50,000 URLs and 50 MB per file; large projects use an index file linking to separate maps by section.

How to split the map so that it earns its keep

Separate files for categories, products, articles and static pages give you diagnostics: in Webmaster you can see which group indexes badly. If 1,200 of 4,000 products are in the index while 39 of 40 categories are, you immediately know where to look. Without the split that information is lost. What to do about a gap you find is covered in the article on indexing.

Priority and change frequency

Search engines effectively ignore the priority and changefreq tags — there is no point spending time on them. The only element genuinely taken into account is lastmod, and only if it is honest. It matters far more that the map is generated automatically when a new page is published, rather than assembled by hand by a content manager once a quarter.

A check after every release

• Open /robots.txt in a browser and read it with your own eyes — 30 seconds that save months
• Test the file in the robots.txt analysis tool in Webmaster against several important URLs
• Make sure the sitemap is reachable, valid and contains the current number of pages
• Compare the URL count in the map with the number of indexed pages — a gap over 20% needs investigating

Who is responsible for this

Experience shows that if these files have no owner, they break at the very first deploy. The right approach is to put robots.txt and the sitemap on the release checklist alongside checking forms and counters. We collected the full list of such checks in the technical audit, and on maintenance projects it is part of regular support work.

Two text files cost less than any other work on a site and yet they determine whether search sees your content at all. Check them today — and if you want an outside view of the whole technical layer, take a look at our approach to SEO and development.