Robots.txt

Use Robots.txt

Robots.txt Help the Search Engines Learn All About Your Website There is a growing interest in the little known file that every website should have in the root directory: robots.txt

It's a very simple text file you can find all about at the robotstxt.org website.

Why should you use it ? Here are some good reasons for you to consider.

Controlled Access to Your Content

With a robots.txt file you can "ask" the search engines to "keep out" of certain areas of your website. A typical area you might like to exclude is your images folder: If you aren't a photographer, painter and your images are for your website use only, there are good chances you don't want them to be indexed and showing up on image search engines, for people to download, or hotlink.

Unfortunately grabbers and similar software (such as Email harvesting applications) will not read your robots.txt file disregarding any indication you may provide in this respect. But that's life isn't it, always someone being disrespectful to say the least ...

You can keep search engines away from content you wish to keep out of sight, but remember your robots file is also subject to attention of hackers seeking sensitive objectives you might inadvertently líst: keeping out the robots while inviting the hackers keep this in mind.

The Growing Importance of Robots.Txt

At SES New York a robots.txt summit was held where major search engines (Ask, Google, Microsoft, Yahoo!) participated, sharing interesting information on this file. Here are some numbers.

According to Keith Hogan from Ask:

i) Less than 35% of websites have a robots.txt file ii) The majority of robots.txt files are copied from others found online

iii) On many occasions robots.txt files are provided by your web hostíng service

It looks like the majority of webmasters aren't familiar with this file. This is going to play a major role as the size of the web continues to grow: Spidering is a costly effort that search engines tend to optimize. Those web sites demonstrating optimal command (which in turn determines efficiency) will be rewarded.