An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file:
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/
But other WordPress blogs have lots of tags included in it.
Also, my XML file's sitemap name looks like this:
How do I add other possible parameters in my robots.txt file? Is the XML file
sitemap.xml.gz name valid, and what is the difference between a
sitemap.xml.gzis a valid name. The
.gz extension just means that it's been compressed (using
gzip compression), so that it's smaller and served faster. Most search engine bots can read gzip'd compressed content.
You can edit the sitemap's XML content just by decompressing
sitemap.xml.gz using a compression utility like
"Robots.txt" is separate from the sitemap.xml, and can be edited with a text editor (without decompressing it). There are many parameters that you can add to the "robots.txt" file, and depends on what you would like to achieve. For more information on possible parameters, Google "robots.txt", or check this link: Robots.txt
XML is the markup language (Extensible Markup Language) and gz (short for GZip) is the compression format in this case. All sitemaps must be in xml format like sitemap.xml or in a compressed xml format like sitemap.xml.gz.
Also your sitemap's url should be the very first line in your robots.txt file and it should read:
or in your case:
The purpose of using G-Zip is to increase your pagespeed which is great for SEO and increasing rank. On a side and similar note, your website should also have a readable page that is accessible to your users called
if you're using html. Here is an awesome link to check and make sure your webserver is using GZip correctly. checkgzipcompression.com