Varnish is a reverse-proxy cache that allows a site with a heavy backend (such as a Drupal site) and mostly consistent content to handle very high traffic load. The “cache” part refers to Varnish storing the entire output of a page in its memory, and the “reverse proxy” part means it functions as its own server, sitting in front of Apache and passing requests back to Apache only when necessary.

One of the challenges with implementing Varnish, however, is the complex “VCL” protocol it uses to process requests with custom logic. The syntax is unusual, the documentation relies heavily on complex examples, and there don’t seem to be any books or other comprehensive resources on the software. A recent link on the project site to Varnish Training is just a pitch for a paid course. Searching more specifically for Drupal + Varnish will bring up many good results - including Lullabot’s fantastic tutorial from April, and older examples for Mercury - but the latest stable release is now 3.x and many of the examples (written for 2.x) don’t work as written anymore. So it takes a lot of trial and error to get it all working.

I’ve been running Varnish on AntiquesNearMe.com, partly to keep our hosting costs down by getting more power out of less [virtual] hardware. A side benefit is the site’s ability to respond very nicely if the backend Apache server ever goes down. They’re on separate VPS's (connected via internal private networking), and if the Apache server completely explodes from memory overload, or I simply need to upgrade a server-related package, Varnish will display a themed “We’re down for a little while” message.

But it wasn’t until recently that I got Varnish’s primary function, caching, really tuned. I spent several days under the hood recently, and while I don’t want to rehash what’s already been well covered in Lullabot’s tutorial, here are some other things I learned:

Check syntax before restarting

After you update your VCL, you need to restart Varnish - using sudo /etc/init.d/varnish restart for instance - for the changes to take effect. If you have a syntax error, however, this will take down your site. So check the syntax first (change the path to your VCL as needed):varnishd -C -f /etc/varnish/default.vcl > /dev/null

If there are errors, it will display them; if not, it shows nothing. Use that as a visual check before restarting. (Unfortunately the exit code of that command is always 0, so you can’t do check-then-restart as simply as check-varnish-syntax && /etc/init.d/varnish restart, but you could grep the output for the words “exit 1” to accomplish the same.)

Logging

The std.log function allows you to generate arbitrary messages about Varnish’s processing. Add import std; at the top of your VCL file, and then std.log("DEV: some useful message") anywhere you want. The “DEV” prefix is an arbitrary way of differentiating your logs from all the others. So you can then run in the shell, varnishlog | grep "DEV" and watch only the information you’ve chosen to see.

How I use this:
- At the top of vcl_recv() I put std.log("DEV: Request to URL: " + req.url);, to put all the other logs in context.
- When I pipe back to apache, I put std.log("DEV: piping " + req.url + " straight back to apache"); before the return (pipe);
- On blocked URLs (cron, install), the same
- On static files (images, JS, CSS), I put std.log("DEV: Always caching " + req.url);
- To understand all the regex madness going on with cookies, I log req.http.Cookie at every step to see what’s changed.

Plug some of these in, check the syntax, restart Varnish, run varnishlog|grep PREFIX as above, and watch as you hit a bunch of URLs in your browser. Varnish’s internal logic will quickly start making more sense.

Watch Varnish work with your browser

The Chrome/Safari Inspector and Firebug show the headers for every request made on a page. With Varnish running, look at the Response Headers for one of them: you’ll see “Via: Varnish” if the page was processed through Varnish, or “Server:Apache” if it went through Apache. (Using Chrome, for instance, login to your Drupal site and the page should load via Apache (assuming you see page elements not available to anonymous users), then open an Incognito window and it should run through Varnish.)

Add hit/miss headers

• When a page is supposed to be cached (not pipe'd immediately), Varnish checks if there is an existing hit or miss. To watch this in your Inspector, use this logic:

sub vcl_deliver {
std.log("DEV: Hits on " + req.url + ": " + obj.hits);
 
if (obj.hits > 0) {
set resp.http.X-Varnish-Cache = "HIT";
}
else {
set resp.http.X-Varnish-Cache = "MISS";
}
 
return (deliver);
}

Then you can clear the caches, hit a page (using the browser technique above), see “via Varnish” and a MISS, hit it again, see a HIT (or not), and know if everything is working.

Clear Varnish when aggregated CSS+JS are rebuilt

If you have CSS/JS aggregation enabled (as recommended), your HTML source will reference long hash-string files. Varnish caches that HTML with the hash string. If you clear only those caches (“requisites” via Admin Menu or cc css+js via Drush), Varnish will still have the old references, but the files will have been deleted. Not good. You could simply never use that operation again, but that’s a little silly.

The heavy-handed solution I came up with (I welcome alternatives) is to wipe the Varnish cache when CSS+JS resets. That operation is not hook-able, however, so you have to patch core. In common.js, _drupal_flush_css_js(), add:

if (module_exists('varnish') && function_exists('varnish_purge_all_pages')) {
varnish_purge_all_pages();
}

This still keeps Memcache and other in-Drupal caches intact, avoiding an unnecessary “clear all caches” operation, but makes sure Varnish doesn’t point to dead files. (You could take it a step further and purge only URLs that are Drupal-generated and not static; if you figure out the regex for that, please share.)

Per-page cookie logic

On AntiquesNearMe.com we have a cookie that remembers the last location you searched, which makes for a nicer UX. That cookie gets added to Varnish’s page “hash” and (correctly) bypasses the cache on pages that take that cookie into account. The cookie is not relevant to the rest of the site, however, so it should be ignored in those cases. How to handle this?

There are two ways to handle cookies in Varnish: strip cookies you know you don’t want, as in this old Pressflow example, or leave only the cookies you know you do want, as in Lullabot’s example. Each strategy has its pros and cons and works on its own, but it’s not advisable to combine them. I’m using Lullabot’s technique on this site, so to deal with the location cookie, I use if-else logic: if the cookie is available but not needed (determined by regex like req.url !~ "PATTERN" || ...), then strip it; otherwise keep it. If the cookie logic you need is more varied but still linear, you could create a series of elsif statements to handle all the use cases. (Just make sure to roast a huge pot of coffee first.)

Useful add-ons to varnish.module

• Added watchdog('varnish', ...) commands in varnish.module on cache-clearing operations, so I could look at the logs and spot problems.
• Added a block to varnish.module with a “Purge this page” button for each URL, shown only for admins. (I saw this in an Akamai module and it made a lot of sense to copy. I’d be happy to post a patch if others want this.)
• The Expire offers plug-n-play intelligence to selectively clear Varnish URLs only when necessary (clearing a landing page of blog posts only if a blog post is modified, for example.) Much better than the default behavior of aggressive clearing “just in case”.

I hope this helps people adopt Varnish. I am also available via my consulting business New Leaf Digital for paid implementation, strategic advice, or support for Varnish-aided sites.