Topics: 

• Drupal Planet
• Performance
• Caching

If your site is left for a while, say throughout the night, and you find that hitting it for the first time in the morning (i.e. a cold start, an initial run) takes longer to load than after clearing your caches, then it's fair to assume that it's not merely a problem of caches expiring and refilling.

To nail this down to a Drupal problem, and not DNS, your Web server or your database, see if it can be reproduced on other systems to eliminate other variables. For example, I'm currently working on a project with Staging and Production each running on a dedicated Gentoo machine, other developers are using Macs for local development and I'm using Ubuntu. We haven't at all coordinated on configuration management, but we're all experiencing the same problem.

In this scenario, there are several things to check:

• Disable the automatic running of Cron. Over at Administration » Configuration » System » Cron, it's set to run every three (3) hours by default. It works by checking the last run on every page hit, and if it hasn't run in that amount of time, it does so. So when your site hasn't been accessed in a few hours, and then a user hits it, he/she is loading the page requested and waiting for a complete Cron run. It's the old Poormanscron that has become a "feature" in Drupal 7. Turn this off by setting it to "Never", and have one of your systems run it in the background every hour. See Configuring cron jobs using the cron command for details.
• Turn off the Update Manager module on Production. You don't want your production server wasting time consulting drupal.org for module updates when Cron runs. Enable it in your development environment so you'll be kept informed of security patches and upgrades. (Ideally, this task would be baked into your DevOps process of periodically syncing your Production database to your Staging/Development sites: turn on Devel, turn off Update Manager, etc..)

Once the above items are handled, there are some things you can do to bring the cache time down as well:

• Increase the Minimum cache lifetime. In Administer » Configuration » Development » Performance, set the Minimum cache lifetime higher if your site doesn't have a lot of writes. If you have a fairly static site, and you have that set to 10 minutes with Cron running every hour, your cache will get destroyed every hour. For situations like this, don't set it lower than "6 hours". You can set it to "1 day" if there are very few content changes.
• Consider warming/priming your cache. Once the cache lifetime expires, the page cache will be cleared. To fill it again, without doing so manually, follow the recipe over at Load Page Cache after cron Runs. Be aware that this is best applied if your Minimum cache lifetime is less than or equal to the time between Cron runs. Otherwise, you'll be hitting your site quite a bit for not too much gain.