There are quite a few caching options out there, and we frequently combine several services to cache every little piece of a site. Think about it: standard Drupal cache, Varnish, Memcache, Redis, various files coming from CDNs, and more.
Sometimes you need to prevent a site, page or everything at a particular path from showing up in Google search. I've heard people say to just disallow the page in robots.txt file. This is actually incorrect.