About a year and a half ago, I came across a post by Miguel Jacq about deploying Drupal automatically with Jenkins. I had grown tired of the manual backup->upload->test->crossFingers->pray->yellExpletives approach to developing or upgrading a Drupal site. Since then, I pretty much can't live without Jenkins (and have also discovered Puppet, which makes rolling new servers extremely easy).

So I implemented Miguel Jacq's approach into my own process and, for several months, found development bliss. It was fun to have Jenkins open in a window and watch the build meter start soon after pushing a commit. When it was done, I had a new platform automatically created through Aegir and the site was automatically migrated (and if the migration failed, rolled back to a working state). It worked well, for a while.

The issue wasn't with the process. The process is sound if you implement a little maintenance. But it just didn't fit my style. It took several minutes, even on a server with SSD's, to do the migration. And when a bad commit was pushed up, that ended up wasting a little bit of time. (Yeah, you're not supposed to push bad commits, but it happens.) And it left me with dozens of unused platforms taking up valuable space.

At this point, I must thank Miguel Jacq for providing the inspiration and the spark to pursue zero-touch deployment methods that fit my style.

While I do much of my development work inside a local virtual machine that closely mimics my production servers (plus utilities like Xdebug and a GUI, though I still do all my coding in Vim), sometimes its inconvenient to pull up the virtual machine to make a small change and the remote development server ends up taking on that role. As in, I want to push a commit and see the results almost immediately. But if this site is being managed by Aegir and you've got Jenkins telling Aegir to do a migration, that 30-second fix can take several minutes to appear. And it really sucks when there was a syntax error that you missed because you did :wq instead of just :w so that you could let Syntastic tell you about it...

But in the same breath, there are definite positives to having Aegir manage your Drupal site. But as with anything, the benefits all depend on your use case. So I've since decided to take another approach (and the scripts are available on GitHub, so fork 'em).

Deploying Drupal with Git and Jenkins (and Drush, of course)

Git and Jenkins both play a central role in my process. Drush should play a central role in any Drupal project. Aegir no longer plays a central role, although it might once again in the future based on what I've seen as possibilities for versions 2.0 and 3.0.

Basically, my development process is like this:

Dev --> Testing --> Code QA --> Staging --> Production

I set up my Git repository with three main branches - Master (which acts as "Production"), Staging, and Dev. Production code is what goes on the site that the public is directed to. It is the stable version. Staging is code that I or other developers believe to be stable and ready for production, but it needs some real world testing and a sign-off by the client. (Before initially going live with a site, this is also where initial content is entered.) Dev is unstable. It's where changes get pushed after not throwing errors on my local environment, but before its been tested with real data. Dev can and will nuke its database at any time.

I use Jenkins to manage this process. What happens is I have a post-receive hook on the project's Git repository that notifies Jenkins when it receives a new commit. In the Jenkins job for deploying the Dev branch, it runs a Fabric script against the Dev server that checks out the code into the correct directory on that server. That means within 10 seconds of pushing a new commit (typically, sometimes a little longer if it was a huge change), that code is ready to go on the remote development server. If for some reason the "git pull" fails on the server (say, for example, that I edited a file directly on the server without committing it), I get an email from Jenkins, and there's a big red ball next to the build in the Jenkins interface, and I can see exactly what was output to the console.

If the project has any custom code, then another Jenkins job is triggered if the Dev deployment was successful. This job runs any automated tests defined for that custom code. This means that I don't have to manually test every scenario after making a minor code change because I took the time to write automated tests. At first it feels like a waste of time to write these, but it pays off in spades pretty quickly. If the tests pass, the ball stays green. If a few tests fail, Jenkins labels the build as "Unstable". If many tests fail or Jenkins sees something I told it to consider a failure, the build is labeled as a Failure. I get an email when the build goes Unstable so that I know that something broke. This usually happens about 10-15 minutes after the code was pushed. (By the way, some of these tests are also checking for code quality and standards. Just because something works doesn't mean that its good code.)

When all of the Testing builds are considered stable and I'm confident that the tests covered what needed to be tested, that's my cue that its probably okay to merge the Dev branch into Staging. This is basically production-ready, but it still needs final approval by the client before it goes Live. The code here is deployed the same way as it is in Dev - a simple job to run a Fabric script against the Staging server to check out the code. It also clears the cache and fires any new database update hooks. At that point, I can manually fire a script that performs a database backup, syncs the database from Production to Staging, clears the cache, fires any new database update hooks, and then the client and I can start testing with real data. The reason that the database sync is done manually is so that changes that hadn't been exported to code (like tweaks to a View) don't get unexpectedly destroyed.

Finally, after the client has confirmed that Staging is ready to go live, the Staging branch is merged into Master and pushed. Once again, Jenkins handles the deployment as soon as it is notified of a new commit.

Now, I'm not telling you that you shouldn't use Aegir. If Aegir is working with your particular development process, then by all means continue using it. For me, once I started using Jenkins to manage other things, it just didn't make sense to have another system handling deployments, too. I definitely learned a lot from Aegir about a proper deployment process using Drush from reading the output of the logs.