robots.txt --- Hide RTL-encoded URLs

by JosephElazar   Last Updated July 25, 2019 05:04 AM

I have a MediaWiki 1.32.0 RTL site (Hebrew) and I desire to hide some of its URLs from search engines like Google and Bing by robots.txt.

The robots.txt command Disallow: /מדיה_ויקי:* can have two UTF-8 versions for RTL languages (Hebrew in this case); one is decoded and one is encoded;

Decoded:

Disallow: /מדיה_ויקי:*

Encoded:

Disallow%3A+%2F%D7%9E%D7%93%D7%99%D7%94_%D7%95%D7%99%D7%A7%D7%99%3A%2A

Both are same in essence - disabling indexation of everything that starts with מדיה-ויקי:.

Which one should I put in robots.txt?

Tags : robots.txt


Related Questions


Updated March 14, 2017 14:04 PM

Updated May 26, 2019 03:04 AM

Updated December 21, 2016 08:01 AM

Updated August 20, 2019 02:04 AM

Updated April 13, 2015 20:01 PM