Sitemaps and robots.txt are always mentioned together, but they are not opposites. One is a list saying “please look here”; the other is a request saying “please do not fetch this”. The second is where the misunderstanding lives — “do not fetch” and “do not show in search” are different instructions.
A sitemap is a list of what you want found
A sitemap is a file listing your addresses. In WordPress it is generated for you, so a marketer never writes one by hand — what remains is checking that its contents are true.
A sitemap is a recommendation, not a command. Being listed does not guarantee indexing, and being absent does not prevent it. But the quality of the list matters: if the addresses you asked to have found do not open, or redirect elsewhere, the whole list becomes less trustworthy.
Because sitemaps are generated automatically, they happily keep advertising addresses you retired months ago. Withdrawn pages, archives you decided not to index, test posts — all of it lingers. Opening the sitemap and reading it once a quarter is a genuinely useful habit.
robots.txt controls crawling, not indexing
This is the heart of the matter. robots.txt tells crawlers “please do not request this path”. It does not say “do not put this in search results”.
Which leads to a peculiar outcome. Block a page in robots.txt, and if anything — another site, or another page of yours — links to that address, the search engine learns the address exists from the link alone. It can then list that address without ever having read it, which is what those results with an empty description or a “no information is available for this page” note actually are.
The trap of doing both
The classic mistake is putting a noindex tag on a page and blocking it in robots.txt. It feels like belt and braces; the result is the opposite. The crawler cannot read the page, so it cannot read the noindex either. If you want a page out of search, leave it readable and let the tag do its job.
One more: blocking the paths that hold your stylesheets or scripts distorts the render stage. That is the third gate from the first part failing — the page is crawled, but what the search engine sees is not what you see.
Indexing policy as a whole continues in the SEO archive, and deciding which addresses stay open and which close is part of the work in our optimization program.
Next part
Having too many open addresses is its own problem. Next: the near-identical URLs that filters and sorting produce, and what a canonical tag actually does.