Get 25% off any hosting plan at Hostgator. Use coupon code wphosting9 at Hostgator checkout
What is robots.txt?
A robots.txt file recommends to bots which files should be accessed and which should not. robots.txt further may be configured to ‘disallow’ access to files and folders by specific bot (e.g. perhaps only Bing, Yahoo and Google bots are wanted to access certain folders). robots.txt is read by search bots and other bots by default and is located at the root of the domain. Be aware that instructions in robots.txt are not enforceable so if a bot is designed to ignore robots.txt then the file will have no effect.
Related posts: