Helpful Robot Tags You Might Use

Writing by Nick Stamoulis on Tuesday, 4 of December , 2007 at 3:39 pm

The natural protocol for search engine robots when they crawl your pages is to index the page and follow all links. All major search engines will follow this protocol unless you tell them not to. Here are some helpful robot meta tags that you might use in order to give specific instructions to the search engines crawling your pages:

meta name=”robots” content=” …” - This meta tag is addressed to all search engines and whatever instructions you give will be followed by all the search engines.

meta name=”googlebot” content=” …” - Only Google will pay attention to this tag.

meta name=”msnbot” content=” …” - Specifies instructions intended only for MSN Live and not other search engines.

meta name=”yahoobot” content=” …” - Instructions specific to Yahoo!

Inside the content=” …” attribute of the meta robot tag, you’ll want to include the following phrases to give specific instructions on how to read, index, or treat content and links on your web pages:

noindex: Tells the search engines not to index a specific page. Your page should not appear in any search engine SERPs if you use this tag with the robots meta. If you use it with googlebot then Google will not index it but other search engines will.

nofollow: This attribute tells the search engines not to follow the links on your page and crawl the pages that you are linking to.

nosnippet: This attribute instructs the search engines to remove the snippets from the SERPs. Whether your snippet (description) is coming from your web page or ODP is irrelevant. This attribute says “Don’t use a snippet at all.”

noodp: This attribute tells the search engines not to use the Open Directory Project (ODP, aka DMOZ) for creating the snippet in the SERPs.

noydir: This is a Yahoo! specific robots tag and tells the robots not to create a Yahoo! titles and snippets in the Yahoo! directory.

noarchive: This attribute tells the robots not to cache a web page or archive it at all.

unavailable_after:[insert date here]: This attribute tells the search engines to remove an indexed page after a specific date.

These robot meta tags can be used in your robots.txt template to instruct the search engines on specific ways to treat your web pages. Be careful in using these and if you want only a specific search engine to respond to your orders you’ll need to include your instructions specifically for that search engine so you wouldn’t use the robots meta tag but the tag specifically for the search engine you want to address.

Category: Meta Tags

• Add this post to Del.icio.us - Digg

No comments yet.