Robot.Txt File Could Be Useful for Your Website

Most of the time, webmasters are not aware of the robot.txt file for their website for SEO. The robots.txt file can be used to instruct the search engine crawlers about what pages on your website should be crawled and indexed on their database, because most website have files and folders that are not relevant or private for search engines to be crawled; therefore, adding a robot.txt at your website root directory improves and makes your website efficiently indexed at any search engine.

A robot.txt file is a simple text file that can be written on the notepad, named as robot.txt. for example, you could write a robot.txt for your wordpress blogging website.

User-agent: *

Disallow: /wp-

Disallow: /trackback

Disallow: /images/

User-agent means that all the search bots (Google, Yahoo Msn ..) should use the following instruction to crawl your website;

Disallow: /wp- means the search bots will not crawl the folder that started with wp-, because most of your private data such like admin file, images are located at wp-admin folder.

Disallow: /images/ means to instruct the search bots not to crawl the folder called as "images".

The robot.txt file should placed at the root directory, otherwise search bots may not able to get those instruction before crawl your website.