A "robot" is a program that search engines use to follow links, read web pages, and create indexes of the information they find on the web pages. There are several reasons why you might want to prevent robots from indexing your web page.
Welcome to Bucaro TecHelp!

Bucaro TecHelp
Maintain Your Computer and Use it More Effectively
to Design a Web Site and Make Money on the Web

About Bucaro TecHelp About BTH User Agreement User Agreement Privacy Policy Privacy Site Map Site Map Contact Bucaro TecHelp Contact RSS News Feeds News Feeds

Victims of Sandy Hook

Stop the Slaughter of Innocents. Congress is bought and paid for by gun lunatics and gun promotion groups. If you want to live in a safe America, help buy Congress back for America. Send a donation to Mayors Against Illegal Guns, 909 Third Avenue, 15th Floor New York, NY 10022

How to Hide From Robots

A "robot" is a program that search engines use to follow links, read web pages, and create indexes of the information they find on the web pages. Also referred to as a "spider" or "crawler", many people think a robot travels around the web. A robot travels the web the same way your browser surfs the web, never leaving the machine it is running on.

There are several reasons why you might want to prevent robots from indexing your web page.

1.You may store administrative information on your web host for backup purposes.
2.You can password protect files without getting involved with the servers authentication mechanisms.
3.You can prevent users from running scripts arbitrarily.
4.You can reduce the bandwidth consumption of your web site.

One way to exclude robots is to place a text file named robots.txt at the root of your web site. In the file, list the robots that you want to exclude, and the files or folders that you want to exclude.

Since it is difficult to know the identities of the robots, you probably want to exclude all robots. The first line of your robots.txt file should be:

User-Agent: *

Below that list the files or folders that you want to exclude. Below is an example of the contents of a robots.txt file that excludes all robots from a folder named "backup" and a folder named "cgi-bin".

User-Agent: *
Disallow: /backup/
Disallow: /cgi-bin/

If you need to exclude many files on a file-by-file basis, you may find a robots.txt file too difficult to maintain. In that case, you may prefer to use meta tag exclusion. A meta tag is an html tag that you place in the HEAD section of your web page. Below is a meta tag that instructs robots to not index the web page.

<meta name="robots" content="noindex">

If you want robots to index the page, but not to follow links on the page, us the meta tag shown below.

<meta name="robots" content="nofollow">

The meta tag shown below instructs robots to not index and not follow links on the page.

<meta name="robots" content="noindex, no follow">

The methods described above allow you control how robots index your web site. Preventing robots from indexing a web page hides it from visitors to your web site. This assumes that you have configured your web site to not allow visitors to "list" the files on your web site. Of course even without the ability to list, a visitor can still access a file if they can guess the name of the file.

RSS Feed RSS Feed


Follow Stephen Bucaro Follow @Stephen Bucaro


Web Design Sections

WordPress Security

90% of business websites have been hacked in the last 12 months. This ebook gives you a complete checklist of things you can do to secure your website without hiring a pricey web developer.

Reader John M Stokes says, "This book is a great resource for security on Wordpress and also does it very quick without having to be a long and boring. Truly worth getting to make sure your Wordpress site is as safe as possible. Click here for more information.

Fire HD
[Site User Agreement] [Privacy Policy] [Site map] [Search This Site] [Contact Form]
Copyright©2001-2016 Bucaro TecHelp 13771 N Fountain Hills Blvd Suite 114-248 Fountain Hills, AZ 85268