Google adds another helpful feature to Webmaster Central that helps Webmasters create a robots.txt file for their sites.

Part two of our article on “Robots.txt best practice guide + examples” talks about how to set up your newly created robots.txt file.

For example, if you make your robots.txt file block crawling during business hours, it's possible that it's cached then, and followed for a day -- meaning nothing gets crawled (or alternately ...