Is Applebot following Crawl-Delay?

by magnetik   Last Updated July 30, 2019 09:04 AM

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down.

Does it follows Crawl-Delay from robots.txt, or not?

Tags : robots.txt


Related Questions


Updated March 14, 2017 14:04 PM

Updated May 26, 2019 03:04 AM

Updated December 21, 2016 08:01 AM

Updated August 20, 2019 02:04 AM

Updated April 13, 2015 20:01 PM