Include "www" in sitemap URL in robots.txt when site redirects to no-www?

by CZorio   Last Updated May 13, 2016 08:01 AM

Website is redirected to remove the www so is redirecting to

Question is do I need to include the www in the sitemap for the robots.txt?

Answers 1

In your situation, Don't include "www" because web crawlers that do read the sitemap directive from your robots.txt will attempt to access the "www" version of your sitemap and then be redirected to the non-www version of the URL.

Not only will you be wasting a couple extra bytes by including "www", but the crawlers that parse the sitemap directive properly will be making two requests to your server: one for the www version, and one for the non-www version.

May 12, 2016 16:02 PM

Related Questions

Updated April 17, 2015 21:01 PM

Updated January 26, 2018 17:04 PM

Updated June 27, 2017 17:04 PM

Updated April 22, 2016 08:01 AM

Updated November 08, 2016 08:01 AM