Understanding common cache-related HTTP response headers

Having a basic understanding of caching is a requirement of being a professional Drupal developer. Unfortunately, there can be many layers of caching which can make it challenging to figure out exactly how best to configure cache settings and troubleshoot potential caching issues.

Web page caching can be thought of as moats around the castle, where each moat is a caching layer and the castle can be thought of as the site's web, database, and other origin servers.

HTTP headers provide a mechanism for the client (usually a web browser) and a server (usually a web server) to exchange information that is not visible to the end user, nor included in the HTML of the page. While there are few types of headers, cache-related headers are generally "response headers".

This blog post isn't meant to be a comprehensive guide to all layers of web page caching, rather we'll focused on common, cache-related HTTP response headers. Information in these response headers can help developers identify which caching layers, if any, were utilized in a given page request. Specifically, we'll be looking at how to view this response header information, and what it can tell us about Drupal page caching and content delivery network (CDN) caching activity on a per-request basis.

Keep in mind that there is not a constrained list of headers that can be added to a page request or response. While there is a common set defined by the Internet Assigned Numbers Authority (IANA) additional proprietary (custom) headers can be added by developers and vendors. It is also important to note that header names are case-insensitive, so the header names "Cache-Control" and "cache-control" are equivalent.

Instructions for viewing cache-related HTTP headers is included at the bottom of this article.

Categories of cache-related response headers

Cache-related response headers can be categorized as either "standard" or "proprietary". Standard headers are those defined by IANA while proprietary headers are added by developers and vendors. Below is a sample of some of the commonly-used cache-related response headers:

Standard (non-proprietary) cache-related response headers

  • Age: the time (seconds) that the object (file) has been cached.
  • Cache-control: general directives for caching mechanisms. For example, if this is set to "no-cache", then the file won't be cached.
  • Expires: the date/time after which the response will be considered stale, and then refreshed from the source. For Drupal sites, by default, this value is set to Dries Buytaert's birth date.

Common proprietary response headers

Again, these are not industry-standard response headers, and different vendors (hosting companies, content delivery networks) may have different implementations. It is best to refer to your specific vendor's documentation for additional details.

  • x-drupal-cache: Drupal-specific header set by Drupal core, values include "HIT", "MISS", and "UNCACHEABLE". Provided by the core Internal Page Cache module, applies only to anonymous users.
  • x-drupal-dynamic-cache: Drupal-specific header set by Drupal core, values include "HIT", "MISS", and "UNCACHEABLE". Provided by the core Internal Dynamic Page Cache module, applies to both anonymous and authenticated users. Allows for partial responses. See blog post by Wim Leers for more information.
  • x-served-by: this header is added by various tools to (normally) indicate the server that provided the response. This includes some (but not all) hosting companies and content delivery networks.
  • x-cache: in general, this indicates if the file was served by a CDN or the origin server. Multiple x-cache values often correspond to multiple "x-served-by" values.
  • x-cache-hits: for some CDNs, this value closely mirrors the "x-cache" value, only with a value of "0" being equivalent to "MISS" and any positive value equivalent to a "HIT" (the value being equal to the number of tries before a hit).
  • cf-cache-status: set by Cloudflare CDN, indicates the status of a cache request. Values include "HIT", "MISS", "DYNAMIC", "EXPIRED", and others. Note that in its default configuration, Cloudflare will not cache HTML - only things like static images.
  • cf-ray, cf-request-id: set by Cloudflare CDN to help trace requests through the Cloudflare network. Used for troubleshooting.

At this point, it is important to note an important difference between two of the more widely-used content delivery networks, Fastly and Cloudflare. Fastly is primarily a content delivery network built on top of the Varnish web application accelerator. While Fastly does have some web application firewall (WAF) capabilities, Cloudflare is both a content delivery network and a full, security-focused web application firewall provider. In some cases (including for DrupalEasy.com), both Fastly and Cloudflare can be used - Fastly for its CDN and Cloudflare for its WAF.

Examples

Below are a couple of examples of cache-related response headers along with a short discussion about what they mean.

Drupal.org cache-related response header values for HTML page
Authenticated user

Cache-Control: no-cache, must-revalidate
x-served-by: cache-sea4481-SEA, cache-mia11376-MIA
x-cache-hits: 0, 0
x-cache: MISS, MISS

Non-cached page served from the drupal.org "origin server" (located at Oregon State University) due to the "Cache-Control" value.

The "x-cache" values indicate that for authenticated users, Drupal.org's CDN (Fastly) did not serve the page. The first "x-served-by" value is the Fastly server closest to the origin server, while the second value is the Fastly server closest to the user loading the page. While "x-cache" indicates that a cached version of the page was not used, it was Fastly's servers that initially processed the request and ultimately retrieved the page from the origin server.

Anonymous user

Cache-Control: public, max-age=900
x-drupal-cache: MISS
x-served-by: cache-sea4420-SEA, cache-mia11369-MIA
x-cache-hits: 1, 4
x-cache: HIT, HIT

The "Cache-Control" value provided by the origin server indicates that the page can be cached for up to 900 seconds (15 minutes). The "x-cache" values indicate that the cached page was served by the Fastly server closest to the user ("cache-mia11369-MIA" in this example).

DrupalEasy.com cache-related response header values for HTML page
Authenticated user

Cache-Control: no-cache
x-served-by: cache-mdw17368-MDW, cache-mia11364-MIA
x-drupal-dynamic-cache: UNCACHEABLE
x-cache-hits: 0, 0
x-cache: MISS, MISS
cf-cache-status: DYNAMIC

DrupalEasy is hosted on Pantheon (which includes the Fastly CDN) and uses the Cloudflare CDN. The "x-cache" values indicate that for authenticated users, Pantheon's default CDN (Fastly) did not serve the page. The first "x-served-by" value is the Fastly server close to the origin, while the second value is the Fastly server close to the user loading the page. While "x-cache" indicates that a cached version of the page was not used, it was Fastly's servers that processed the request and ultimately retrieved the page from the origin (Pantheon) server.

The "cf-cache-status" of "DYNAMIC" indicates that the file is not eligible for caching by Cloudflare. This is normally a result of a directive from the origin server to never cache the file ("Cache-Control: no-cache").

Anonymous user

cache-control: max-age=60, public
x-served-by: cache-mdw17325-MDW, cache-mia11383-MIA
x-drupal-cache: HIT
x-drupal-dynamic-cache: MISS
x-cache-hits: 0, 1
x-cache: MISS, HIT
cf-request-id: 0775a92a710000e958291b2000000001
cf-ray: 60cfaaf0aa52e958-MIA
cf-cache-status: DYNAMIC

For an anonymous user, the "x-cache" value of "MISS, HIT" indicates that the 2nd "x-served-by" value (the Fastly server close to the user) provided a cached response.

DrupalEasy.com cache-related response header values for image (site logo)
Anonymous and authenticated users

Cache-Control: max-age=31622400
Expires: Fri, 31 Dec 2021 17:54:40 GMT
x-served-by: cache-mdw17349-MDW, cache-mia11320-MIA
x-cache-hits: 1, 1
x-cache: HIT, HIT
cf-cache-status: HIT

The "cf-cache-status" value of "HIT" indicates that Cloudflare served this image. The "Cache-Control" value indicates that the image can be cached for up to 3162240 seconds (

Manually "bust" the cache

During development, sometimes it is necessary to "bust" the cache - that is, to request an uncached version of the content. While this can be done by clearing/purging/rebuilding caches at every level of the site's infrastructure (Drupal, Fastly, Cloudflare), it can also be done by adding a unique querystring variable to the content as they are cached according to their unique URL, including querystring variables. So, if you want to request an uncached version of "logo.png", simply append a meaningless querystring variable to it - for example, "logo.png?buster=2358973430985234895".

Summary

As the previous examples show, in order to be able to make sense of cache-related response headers, it is important to know which, if any CDNs are being used as well as which, if any, Drupal-level caching modules are enabled.

Resources

Viewing cache-related HTTP headers

All major browsers provide relatively easy access to HTTP header information, if you know where to look. Here's a quick guide:

Safari

  • Open Web Inspector ("Inspect Element" anywhere)
  • Go to "Network" tab.
  • Reload page.
  • Click on (usually) first entry in list of files - this will be the main HTML file for the page.
  • Click on the "Headers" sub-tab.

Firefox

  • Open the network tools via the "Tools|Web Developer|Network" menu item.
  • Reload page.
  • Click on (usually) first entry in list of files - this will be the main HTML file for the page.
  • Click on the "Headers" sub-tab.

Chrome

  • Open the network tools via the "View|Developer|Developer Tools" menu item.
  • Go to "Network" tab.
  • Reload page.
  • Click on (usually) first entry in list of files - this will be the main HTML file for the page.
  • Click on the "Headers" sub-tab.

Edge

  • Open Developer Tools (Crtl-Shift-I).
  • Go to "Network" tab.
  • Reload page.
  • Click on (usually) first entry in list of files - this will be the main HTML file for the page.
  • Click on the "Headers" sub-tab.

Thanks to Andy Giles from Blue Oak Interactive for reviewing this blog post.

Original Article: