What Is A Robots.txt File
A Robots.txt file is a method to provide search engine robots information about what areas of your site should be accessed and what areas should not.
A search engines robots sometimes referred to as ”bots’ or spiders crawl around the internet searching for sites to index. The first thing that they usually look for is a Robots.txt file. This file contains a link to every page that you want a search engine to index.
It follows the following process – a robot wants to visit a site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /
The User-agent: * means this section applies to all robots. The Disallow: / tells the robot that it should not visit any pages on the site.
Details Of A Robots.txt File
There is no detailed standard, however there is what is called the The Robots Exclusion Protocol which utilize two specific lines – User-agent and Disallow.
- User-agent: * – this section applies to every robot. It like a wildcard.
- Disallow: / – excludes all robots from the entire server.
- Disallow: /bin/ – excludes all robots from part of the server.
Examples of a Robots.txt File
To exclude all robots from the entire server
User-agent: *
Disallow: /
To allow all robots complete access
User-agent: *
Disallow:
To exclude all robots from part of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
To exclude a single robot
User-agent: UnwantedBot
Disallow: /
Creating a Robots.txt File and adding it to your site can help your pages get indexed faster and more regularly by the various search engines than if you had nothing. A Robots.txt file also gives you more power over what you would like the search engines to find and what you would not want indexed.
Cemil Ozcelik
www.OnlineBusinessBlackbook.com © 2009 - 2010
If you like this post, you might also like:
If you found this page useful, why not use the social media buttons above to add it on your favorite social sites and also
subscribe to the RSS feed for more interesting posts from Online Business Blackbook.
Also Check Us Out On Facebook and
Twitter
{ 1 trackback }