The robots.txt file is not a HTML file it’s a simple text file that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters uses the robots.txt file to help the search engines index the content of their websites. All major search engines support the basic functionality it offers. There are some extra rules that are used by a few search engines which can be useful too.