How we installed a Drupal security patch on 1300 sites, stress-free!
04 Apr
How we installed a Drupal security patch on 1300 sites, stress-free!
Nick Veenhof
Drupal
Yesterday a highly critical security issue in Drupal was released. The issue itself is considered critical, because, the way we understood, it makes it possible to execute code as an anonymous user. This could lead to a complete hack of your site and complete exposure of your content - or, worse, if your webserver is badly configured, a full-scale hostile takeover of your server. (More background info available here and here.)
The issue was announced to the Drupal community a week early, so our Dropsolid team had plenty of time to anticipate and prepare. Currently, Dropsolid serves 482 unique and active projects, which contain on average three environments. To be more precise, this gave us a whopping 1316 active Drupal installations to patch. These environments are located on 65 different servers. 45 of those servers are out of our hands and are managed by other hosting companies, such as Combell or even dedicated hardware on site with the customer. At Dropsolid we prefer to host the websites within our own control, but to the Dropsolid Platform this ultimately makes no difference. For some customers we also collaborate with Acquia - these clients are taken care of by Acquia’s golden glove service.
So, back to preparing to patch all the different Drupal installations. We would be lying if we said that all Drupal installs were running on the latest and greatest, so we used Ansible and the Dropsolid Platform to gather all the necessary data and perform a so-called dry run. This was a real-world test across all our installations to verify if we could pass on a patch and then deploy it as soon as we have confirmed that the patch works for all the versions that we have available on our Dropsolid Platform. For example, it verified if the patch tool is available on the server, it injected a text file that we then patched to make sure the flow of patching a Drupal installation would go smoothly, etc. Obviously we detected some hiccups as we were testing, but we were left with enough time to resolve all issues in advance.
Throughout the evening, we had plenty of engineers on stand-by, ready to jump in should something in the automated process go wrong. The entire rollout took us about 2 hours - from the release of the patch over verifying the patch on all the different Drupal releases to rolling it out on all sites and, finally, relax with a few beers. This doesn't mean we had it easy. We had to work a lot, but a lot of hours just to make sure we could handle this load in this amount of time. That is why we are continuously building on our Dropsolid Platform.
Those who joined our hangout could bear witness to exactly how comfortable and relaxed our engineers were feeling during the rollout.
You might ask, joined our hangout? What are we on about exactly? Well, since the Drupal community was in this together, I suggested on Twitter to all join in together and at least make it a fun time.
A few nice things that happened during this hangout:
- Someone played live ukelele for us while we waited
- Someone posted a fake patch and made everyone anxious, but at least it was a good test!
- People were able to watch Dropsolid in total transparency how we coped with this patch and were also able to interact and talk to others in the hangout.
It made the whole evening a fun activity, as witnessed by Baddy Sonja.
Obviously this couldn’t have happened without the help of our great engineers at Dropsolid - and also because we invest a lot of our R&D time into the development of the Dropsolid Platform, so we can do the same exercise times 10 or times 100 without any extra human effort. Thanks to the Drupal security team for the good care and the warning ahead of time. It made a tremendous difference!
All our Dropsolid customers can rest assured that we have their backs, all the time!
If you are not a Dropsolid customer yet and you are interested to see how we can help you make your digital business easy, we’d be more than happy to talk. If you are running a Drupal site and need help with your updates or with your processes, we’d be glad to to help out and onboard you onto our Dropsolid Platform. You can keep your server contract while benefiting from our digital governance and expertise. Are you in charge of many many digital assets and feeling the pain? Maybe it’s time you can start doing the fun things again - just have a chat with us!