Every website on the internet should have a robots.txt file located in the root directory.
You may use the form below to create your own fairly extensive robots.txt file automatically. Simply read and select which bots or options you would want your robots.txt file to contain. When you are finished, simply copy and paste into notepad, name it robots.txt and upload it into the root directory of your domain and you'll be good to go.
A robots.txt file helps various robots (internet parsing utilities) to know which part of your domain to parse and which parts not to parse. There are hundreds of internet parsing agents, or robots on the world wide web, so it's important to give direction to those robots that actually do read the robots.txt file.
Windows 98, ME, 2k, XP, XP-pro, Vista
MS-DOS & NT Platforms
Computer Hardware Repair
Hard Drive, CD-ROM, DVD, Motherboards, Power Supply, PCI Cards, Wireless Installations
Save the unit you have by giving it the maintenance it needs. From parts and accessories to complete rebuilds.
Onsite Repairs
9am to 3pm 7 days a week.
(Vidor, Port Arthur, Lumberton, Beaumont area only)
WebshoppeSolutions - Robots.txt Generator Tool
Not all bots and parsing utilities online will follow the robots.txt exclusion protocol. This robots.txt, however, covers most or all of the major ones found online that might read robots.txt.
In order to block parsing agents that ignore robots.txt, we may have to disallow them through .htaccess.
Link back to us if you find this tool to be useful.
