Yellowpipe Internet Services

robots.txt Generator

Create your own robots.txt

A robots.txt file is a short text file that resides in your home directory. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see. This freeware utility will assist you in the creation of robots.txt files.

As a little 'extra' it also includes a block to many unwanted spiders that only crawl your site in order to collect the email addresses stored on your pages. (SPAM-bots)

Fill in the fields replacing the default values with your own ones. Create your robots.txt file, save it on your desktop and upload it at the root of your site.
Example: http://www.domain.com/robots.txt

 

   Your website URL

Disallow all robots:
   Your site won't be cralwed by any robots including the safe ones like google.
[If checked -> click submit now]

Directories or single pages to exclude:
   for http://www.yoursite.com/cgi-bin/   enter  /cgi-bin/

   for http://yoursite.com/images/   enter  /images/

   for http://www.yoursite.com/free/tibet.html   enter  /free/tibet.html

   for http://subdomain.yoursite.com/private/   enter  ../subdomain/private/

   for http://www.yoursite.com/other/   enter  /other/

   for http://www.yoursite.com/other/   enter  /other/

Disallow unsafe robots:
   Include our sorted list of 135 additional unsafe robots that crawl your site for spamming or other dirty activity.

 

Powered by Yellowpipe Also check out other free webmaster tools.

 

 

About Us | Site Map | Privacy Policy | Contact Us | Yellowpipe RSS News Feed ©2001-2007 Yellowpipe.com