Preventing Robots From Crawling Your Pages

Writing by Nick Stamoulis on Saturday, 10 of November , 2007 at 4:13 pm

There might be a time when you do not want the search engines to crawl certain pages. You can prevent them from crawling certain pages by simply inserting the right code in its proper place. Contrary to some people’s thinking, you wouldn’t use the “no follow” tag.

The no follow tag is used to tell the search engines not to transfer your PageRank to the pages your links connect with. That’s a different thing altogether. The search engines will still crawl those links and index the pages you link to. But if you don’t want them indexing the pages on the other side of those links what would you do?

The proper code is a robots meta tag. It’s called Robots Extension Standard. The robots meta tag uses the following syntax:

meta name=”robots” content=”noindex,nofollow” /

Don’t forget to add the opening and closing brackets (< and >) for this sentence. If you’ll add this code within the HEAD section of your HTML, the search engines will not index that page and will not follow the links. To prevent following links only, take out the noindex attribute. To prevent the indexing of the page only then sans the nofollow attribute.

Category: Robots

• Add this post to Del.icio.us - Digg

No comments yet.