Prevent XML sitemaps from showing up in Google search results

by Stephen Ostermiller   Last Updated June 26, 2015 12:17 PM

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query:

sitemap in search results

I don't understand why Google would choose to show sitemap files in search results to begin with. These files are not meant for human consumption.

Google needs to be able to crawl it to be able to process it, so I can't disallow it in robots.txt. I just don't want them to put it into the search results after processing it.



Answers 4


put the following into the robots.txt

User-agent: *
Disallow: /sitemap.xml

instead submit your sitemap through google webmaster tools.

konghou
konghou
May 21, 2014 07:45 AM

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed.

The recommended method to prevent these files from being indexed by Google is to include an X-Robots-Tag HTTP response header when serving the XML sitemap. For example:

X-Robots-Tag: noindex

Just like including a robots META tag in HTML files, the X-Robots-Tag header can be used for any type of file.

Reference: This document (from Nov 2008!) appears to quote our very own John Mueller (Google) with regards to the use of the X-Robots-Tag response when dealing with XML sitemaps.
Yes, Google Will Index & Rank Your XML Sitemap File

Update 09-2015: For more information see Google's developer guide:
Robots meta tag and X-Robots-Tag HTTP header specifications

w3dk
w3dk
May 21, 2014 08:15 AM

w3d's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so:

<Files ~ "sitemap.*\.xml(\.gz)?$">
  Header append X-Robots-Tag "noindex"
</Files>

Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?

Stephen Ostermiller
Stephen Ostermiller
May 21, 2014 17:10 PM

Why does it matter?

If you can actually find your sitemap in SERP then you have bigger problems.

I would focus more on getting pages up with useful content instead. That way, you will have a very hard time even finding you sitemap. Not that you would care at that point anyway.

P.S.

Pretty much every one keeps sitemaps in the same place. So if someone wanted to find where you keep it they will :)

dasickle
dasickle
May 22, 2014 19:20 PM

Related Questions


Updated September 19, 2018 07:04 AM

Updated April 15, 2016 08:01 AM

Updated June 21, 2017 02:04 AM

Updated December 15, 2017 05:04 AM

Updated September 15, 2017 09:04 AM