Override robots.txt Exclusion

by Kohjah Breese   Last Updated September 01, 2019 02:04 AM

I use robots.txt to block all robots.txt compliant bots from a directory like this:

User-agent: *
Disallow: /sub/*

How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?



Answers 1


Adding this at the top of the robots.txt works:

User-agent: Mediapartners-Google
Disallow:

N.B. you may need AdSense to download your robots.txt again, which I believe they do on a 24hr basis.

Kohjah Breese
Kohjah Breese
September 01, 2019 01:54 AM

Related Questions


Updated May 26, 2019 03:04 AM

Updated December 21, 2016 08:01 AM

Updated August 20, 2019 02:04 AM

Updated July 02, 2015 13:01 PM

Updated May 20, 2016 08:01 AM