How to Edit or Remove robots.txt on a WordPress powered website

by Facet   Last Updated January 27, 2017 14:01 PM

When using Google's fetch test on my WordPress site, it reports that my robots.txt is blocking pages and resources from being indexed.

Google reports that my Robots.txt looks like this:

User-agent: *
Disallow: /

How can I resolve this issue?

  • Where can I find robots.txt?
  • How do I edit or remove this file?


Answers 3


You need to remove both lines from your robots.txt file. The robots file is located in the root directory of your web hosting folder, this normally can be found in /public_html/ and you should be able to edit or delete this file using:

Simon Hayter
Simon Hayter
January 25, 2017 14:24 PM

You have to update robots.txt file in WordPress through Cpanel because When you install Yoast SEO plugins, It automatically generates robots file for your site. But i you really want to edit it with your own instructions, login your hosting server->file manager then edit it.

Gary St Clare
Gary St Clare
January 27, 2017 07:08 AM

If this is fresh install of Wordpress it could because you have set the privacy settings to stop search engines from crawling the site.

(and there will be no physical robots.txt on the server, as wordpress creates it on the fly)

Go to your settings in wordpress and see if this box is ticked:

enter image description here

If so, uncheck it and then the robots.txt should change to

User-agent: *
Disallow: 

If you still have issues and the robots.txt is still set to block crawlers, then explore the other option as outlined by Facet.

Max
Max
February 16, 2017 09:34 AM

Related Questions


Updated November 22, 2018 04:04 AM

Updated May 21, 2019 06:04 AM

Updated July 05, 2018 14:04 PM