robots.txt --- Hide RTL-encoded URLs

by JosephElazar   Last Updated July 25, 2019 05:04 AM

I have a MediaWiki 1.32.0 RTL site (Hebrew) and I desire to hide some of its URLs from search engines like Google and Bing by robots.txt.

The robots.txt command Disallow: /מדיה_ויקי:* can have two UTF-8 versions for RTL languages (Hebrew in this case); one is decoded and one is encoded;


Disallow: /מדיה_ויקי:*



Both are same in essence - disabling indexation of everything that starts with מדיה-ויקי:.

Which one should I put in robots.txt?

Tags : robots.txt

Related Questions

Updated March 14, 2017 14:04 PM

Updated May 26, 2019 03:04 AM

Updated December 21, 2016 08:01 AM

Updated August 20, 2019 02:04 AM

Updated April 13, 2015 20:01 PM