How to scale content delivery infrastructure by implementing Edge Side Includes in Drupal.

Developers and webmasters who oversee websites with millions of users need to provide a solution to keep their infrastructure from getting overloaded with requests. Scaling up web and database servers is one option, but it can be costly and inefficient. Instead, people have increasingly turned to a Content Delivery Network, or CDN, as a type of protective layer in front of their web and database servers.

What does a CDN do?

The CDN provides a cached layer of content close to the user, often referred to as “the edge.” When a user requests a homepage, for example, they are directed to the cached static version of that page on the CDN rather than overloading the web server or accessing the database, thereby scaling content delivery.

Illustrating scaled content deliver supporting millions of requests while only passing on a small percentage of those requests to a web server, which in turn makes even fewer requests to a database server.

The CDN serves static content. So, what happens when web content is updated? There are a few different options here:

• The cache can be programmed to expire after a certain number of hours or days.
• Cache entries can be proactively purged when updates are made.
• Changes to individual page content can be fetched as each page is requested by a user.

More problematic, however, are the changes that affect each and every page. For example, a global banner or a menu often changes the header or footer. This is where our recent work with Edge Side Includes comes in.

Edge Side Includes

Edge Side Includes (ESI) is both a web standard and an XML-based language that enables the dynamic generation of HTML pages at “the edge”. We recently worked with one client to solve the problem: How can we enable our cached content to remain fresh even after we make updates to some of the global parts?

We used Drupal to solve this problem by generating and rendering this global content as ESI fragments. These ESI fragments could then be included by all of our client’s web properties by using ESI include statements, regardless of how they were built.

Illustrating pseudocode of static markup on a web page, where each web page must include that markup and therefore each page must be retrieved from cache, the web server, and may require a shorter cached lifetime.Illustrating pseudocode of ESI includes and their corresponding ESI fragments indicating that now each web page may be able to have a longer cache lifetime since the ESI fragments are referenced and can have their own cache lifetime.How ESI Works

For our client, we developed a way to render ESI fragments in Drupal that included page parts (specifically the header and footer) on non-Drupal sites so that when those page parts change in Drupal, the non-Drupal sites get those changes automatically. To accomplish this, at the cache layer we have all of these pages that have unique page content and then the same content on each page for including the header and the footer.

Using ESI fragments, if a request is made for a page, the first thing that happens is a request for the header content, and then for the cached footer content. Now if something in the header is changed, only one page needs to be updated.

Drupal and Edge Side Includes

The use case that we solved here was specifically for the header and the footer, but our client wanted to have similar branding across all of their web properties, and they wanted it to be governed by the content management that's done with Drupal. Drupal can do this by rendering the actual ESI fragments at two different endpoints.

Our custom module defines two routes, one for the header and one for the footer. Our controller maps to both of those routes and returns an empty render. Then, in the dot module files, we made sure that we're only including the sort of meta tags that we need and the libraries that we want. In our theme, we have special templates for the ESI fragments. We also made sure that we leveraged some of the core functionality by still going through the render API.

We're rendering blocks, we're rendering menus, we're using libraries, and we're respecting cache tags.

Illustrating the breakdown of responsibilities between Drupal modules (route definition, controller creation, page attachment alters), themes (templates, library definition), and core (render process for various entities, libraries, and cache api).

Our client developed page templates that use business logic to set whatever variables might be needed to deliver their page content, adding our ESI include statements to actually grab the header and the footer content. They’re hosting all of their non-Drupal pages on a server that will provide this ESI service. Our client also determines the cache lifetime, both for their page templates and for the ESI fragments that Drupal's actually hosting.

Illustrating the breakdown of responsibilities for ESI implementers/consumers (business logic, variables parameters for fragments, page templates with variables and includes) and service providers (cache lifetime configuration, ESI fragment routes, ESI service / hosting).Personalization Through ESI

So, what’s next for Drupal and ESI? Another implementation that we're using in Drupal with ESI is a content model where a URL can reference internal or external endpoints and then include content from a URL that references static assets. This is CSS that we can include for personalization. This will allow us to do things like get data from Google tag manager or from Marketo or Mailchimp or a similar platform and make some decisions about the route, which could be a view page. Then, we could dynamically write the ESI include source based on the content they want to render personalized. We’ll let you know as we progress!

Development
Drupal