The robots.txt file is not a HTML file it’s a simple text file that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters uses the robots.txt file to help the search engines index the content of their websites. All major search engines support the basic functionality it offers. There are some extra rules that are used by a few search engines which can be useful too.
If webmasters can tell the search engines spiders to skip pages that they do not consider important enough to be crawled, then they have a better opportunity to have their most valuable pages featured in the search results pages. The robots.txt file is a simple method of essentially easing the process for the spider to return the most relevant search results.
Making an empty robots.txt file is simple, just open up a text editor such as notepad, save the blank file as robots.txt or you can add this bit of code before you save it:
User – agent *
Disallow:
That bit of code tell all search engines that are welcome to crawl all areas of the website, “user-agent” means what search engine bits the instructions are for, the star (*) that comes after it means the instructions are for all bots. “Disallow:” means what parts of the website are not to be crawled, because there is nothing after it, bots know they can crawl everything. Be sure to keep the code on separate lines.
There are a couple of things that always need to be remembered when managing robots.txt records and these include:
· The filename is case sensitive; please remember to use robots.txt and not Robots.txt.
· The record is publically accessible and anyone has access to the file. It’s important to apply better security to sections of your website you are trying to keep hidden for security reason.
· A single passage of prohibit is permitted per URL.
· Along with the document the “no index, take after” tags should also be used on every related page.
This also increase spider ability for the search engines. Obviously, despite the fact that this a small aspects of the search engine optimization process, if used correctly, a robots.txt can be a significant benefit.