The RobotsEditor is a Manila Plugin that allows Managing Editors to add and edit robots.txt files to their Manila hosted domains. I originally created the plugin to make sure that robots and spiders were not visiting my development sites.
When I started hosting Manila sites for others, I added editing capabilities for Managing Editors. Everything was working fine until UserLand introduced their own robots.txt editor. The two solutions did not play nice together. UserLand's code overwrote my storage location (*manilaWebsite.["robots.txt"] seemed like a good idea at the time). My code, which added its own responder that bypassed UserLand's solution. The results were not pretty.
My first revision just changed my storage location. This helped, but it still wasn't good solution, because it bypassed UserLand's default (which disallows access to the referer page).
My latest revision fixes that problem. I am still using a responder that bypasses UserLand's robots.txt solution, but it now defaults to UserLand's default. So even though it is using separate responder from Manila, it still serves the same robots.txt file.
The contents of the robots.txt, depends on whether a Managing Editor has enabled the plugin or not. If they haven't then the robot receives the same results as UserLand's editor would produce. If they have, then robots receive what the Managing Editor specified using RobotsEditor.
So why use the RobotsEditor plugin, if it provides the same functionality as UserLand's robots.txt editor. The RobotsEditor plugin provides greater flexibility. UserLand's editor only allows you to add paths. You cannot edit the User-agent, so you can not allow access to some spiders and robots and not others. RobotsEditor, on the other hand, is a freeform editor. You can add anything you like, specifying both User-agents and disallow fields. Also since the RobotsEditor stores the robots.txt file as a string it is served much faster than Userlands robots.txt file which is created on the fly.
If you are using the RobotsEditor plugin, you may want to know a little more about the robots.txt specification. The editor page provides links to some of this information.
1:06:01 PM 
|
