If you have a public-facing development site that you don't want search engines to stumble upon and start indexing, here's a method that you can try on your Drupal site (or any other Apache site for that matter). Go through the usual process of adding your normal robots.txt file to your site, but then create an alternate robots.txt file (let's call it "robots-dev.txt") that's specific to the dev site and tells search engines NOT to index your content. Here are the contents of the text file:

<br>#<br># robots.txt<br>#

read more