Development Environment for Drupal

There are virtually an unlimited amount of ways to set up a development environment, and it all depends upon your project’s individual needs and your preferred workflow. Sometimes, though, it just takes experience and exposure to other ideas to figure out what would be best. So, here is a description of my development environment.

Separation of workspaces

Separating workspaces is vital. I’m not sure I can stress this enough, but you should never work directly on your production website. In fact, I even limit available permissions on the production site to prevent anyone doing development work on it. Do not risk exposing your visitors to development related bugs.

I have at least three workspaces, even on small sites where I am the only developer. The minimum workspaces are:

Production
Staging
Development

Production is for visitors, user data, and adding, deleting and modifying content. It is not for adding modules, changing content fields, modifying blocks or fixing bugs!

Staging is the place for a test deploy from development before deploying to production. When using my database scripts, I perform a merge of the production and development databases here and ensure that it is working correctly.

I will also use staging to show changes to clients for final approval. However, no changes created in staging is preserved. It’s purely for observation and testing.

Development is for all the nitty gritty work. Have at it, and don’t be scared to break anything! I frequently dump and restore the database while developing so I can quickly get rid of any bad configurations and work with a stable version.

When a team grows larger, I add in a test workspace and divide the development section into sandboxes. For example:

Production
Staging
Test
Development
- sandbox-kathleen
- sandbox-joe
- sandbox-jane

Each developer works in their own sandbox so no one is stepping on each other’s toes. Test will have the latest version of development which includes all of each individual’s work. The testing team then works with the test workspace to find any bugs and report them back to development.

Where should each work space be located?

The best practice is to keep production away from development on a server level. However, for small sites with low server load, this is not as much of a concern and each workspace could be located on the same server. In fact, this site and several of my other low-load sites are all located on the same Dreamhost account. Yet, HarvardScience’s production site is located alone on its own dedicated server.

When I do separate them, I have production and staging on the live server. This allows me to test the new version of the site on the same environment that production is located. Each developer’s sandbox could be located on the same development server or on their own computers.

Protect your workspaces from prying eyes

If your workspaces are on a publicly accessible URL, you should password protect them. Trust me, obscurity is not security. Google will find it, index it, and people will come. I still have a lot of holes to cover up due to not password protecting my development sites.

The best way to password protect them is a browser level password so that the access protection doesn’t interfere with the site’s configuration. Using the command line, create the file to store usernames and passwords:

htpasswd -c Filename username

Then add more usernames using:

htpasswd Filename username2

Once you have created that file to store your usernames and passwords, you can add the following to each workspace (except production) to your .htaccess file:

AuthUserFile /path/to/password/file/.Filename AuthType Basic AuthName "Protected" Require valid-user

Migrating workspaces

I use a version control system to manage my migrations. I am currently using subversion (SVN), however my company recently switched over to using git. After using git for one day, I wished all of my other sites were using git; I loved it that much.

For simple sites, including this blog, I don’t bother managing release branches and tagging. Both production and development are pointing to trunk, and I just update production when trunk is stable.

However, for larger sites that include longer development time between deployments, I create tags and release branches. My subversion repository looks like this:

- trunk - branches   - reworking_template   - dev_branch2   - dev_branch3 - tags   - 2007-10-27   - 2007-12-04   - 2008-01-25 - releases   - 2007-10-27   - 2007-12-04   - 2008-01-25

Development points towards trunk, unless a developer creates a branch for some experimental work, or something that will take a long time. Branches are generally for avoiding critically breaking trunk and preventing other developers from being able to do their work.

Tags are a snapshot of trunk for some interesting moment you’d want to refer back to. Generally I just use it for noting an exact state of the site as it was released. Tags are considered static, and no more commits should be posted to them.

Releases are more like branches, recording the branching production is taking away from trunk. My back up system is to dump the production database and commit it and all changes to the files directory every evening. These backup commits all get posted to the current release branch. I decided to go this path as it would make it much easier to grab production data and use it during development, as well as quickly being able to go back to a last-known-good-state if anything became corrupted on the production site.

Migrating from one workspace to another for me is then just running svn update, or pointing the space to the new release branch. Pretty easy.

Sure, you could copy files and such, and keep directories with snapshots over time; but really, use version control. Seriously. Even if you’re a one-man-show, use version control. You can take out changes from amongst your other changes and you can find exactly when bugs were introduced. Once you use version control, you’ll never go back.

Larger sites may want to consider implementing deployment systems. My company uses Capistrano to deploy Fusecal to our production and staging servers.

Migrating database changes

I wrote up the database migration system I use in a previous post. In short, it allows me to merge the production data with development. As long as you dump the database in a fashion that is diffable, and ignore certain data that is unnecessary to store in development (like watchdog), you can actually use your version control system to merge in changes from developers. With good development practices, conflicts should be rare and easy to fix if they do occur.

Other companies focus on creating modules to do a lot of the heavy lifting, and/or utilize .install files for their changes so they get performed when running update.php. The devel module has a means for recording form submissions, which you can then export and replay in the next workspace. And lastly, many will write down every database related change and perform them again in the next workspace.

Whatever method you use, I encourage you to consider a method you can do quickly and reliably. If it’s too slow you won’t do it and reduce testing opportunities. If it isn’t reliable, you’ll risk losing steps and making mistakes. Be careful.

Managing differences between production and development environments

There are some settings you want in production, but not development. For example, aggressive caching is a huge detriment in development, and you also don’t want the password protection mentioned above to appear in production. Not to forget the fact that they’ll also be pointing to separate databases! (At least they should, otherwise you could be causing performance issues for production)

To solve this, I do not include .htaccess and settings.php in my version control. I use svn ignore on those files, and instead version control copies of them, named .htaccess.example and settings.php.example. I then include commented-out code. When I create a new working space, I copy the example files to the original filename and uncomment out any necessary code for that working space, and point settings.php to the correct database.

Anything located in the ‘variables’ table can be overridden in your settings.php file. Open up settings.php and scroll all the way to the bottom and read the section labeled ‘Variable overrides’. It’s here that you can set up various settings that are appropriate for production, but inappropriate for development.

Here is what I use for this blog:

$conf = array(   'cache' => '1',   'cache_lifetime' => '0',   'error_level' => '0',   'preprocess_css' => '1',   'watchdog_clear' => '1209600',   'javascript_aggregator_aggregate_js' => '1',   'javascript_aggregator_optimize_js' => '1',   'xmlsitemap_engines_ask_submit' => '1',   'xmlsitemap_engines_google_submit' => '1',   'xmlsitemap_engines_live_submit' => '1',   'xmlsitemap_engines_yahoo_submit' => '1',   'xmlsitemap_engines_ask_url' => 'http://submissions.ask.com/ping?sitemap='.$base_url.'sitemap.xml',   'xmlsitemap_engines_google_url' => 'http://www.google.com/webmasters/tools/ping?sitemap='.$base_url.'sitemap.xml',   'xmlsitemap_engines_live_url' => 'http://webmaster.live.com/ping.aspx?siteMap='.$base_url.'sitemap.xml',   'xmlsitemap_engines_yahoo_url' => 'http://search.yahooapis.com/SiteExplorerService/V1/ping?sitemap='.$base_url.'sitemap.xml',   'xmlsitemap_submit' => '1',   'xmlsitemap_update' => '1', );

As you can see, I use aggressive caching, hide error messages, and set up xmlsitemap for the production site. All of which I don’t want happening in development, so I keep this section commented out when in a development workspace.

Managing the filesystem

Drupal core shouldn’t be touched, so I decided to just pull it out! This gets it out of the way, and prevents that momentary confusion where some developers would impulsively go to ‘/modules’ or ‘/themes’ instead of ‘/sites/all/modules’ and ‘/sites/all/themes’. In the end, everything that is specific to your site is all in one place and separate from the general Drupal files, making it easier to upgrade.

Within a version control environment, it allows you to keep Drupal external from your site’s files, relieving the need to duplicate Drupal’s core files for each site. I have a separate section for tracking changes to Drupal, and I use svn externals for pointing to Drupal’s files. When managing multiple sites, this allows me to update Drupal’s files once, and then change the tag of the Drupal release that each site points to.

You can setup externals with:

svn propedit svn:externals .

This will popup an editor where you enter your external settings. Mine are:

drupal http://svn.domain.tld/webdev/drupal/tags/Drupal-5.7 dbscripts http://svn.domain.tld/webdev/dbscripts/trunk

Changing what your external points to is just a matter of changing that one line. For me, when Drupal 5.8 comes out, I’ll tag the release in my repository then change the character ‘7’ to an ‘8’ and I’m all done!

Instead of putting the ‘files’ directory in Drupal-root, I put it under ‘/sites/default’. The ‘sites’ folder is pulled out and put in the same level as the Drupal folder, and a relative sym link within the Drupal directory will look for a ‘sites’ folder one level down. You can create the sym link via:

cd /your/path/to/drupal ln -s ../sites sites

The web root is pointed at the Drupal folder. The web root is the bottom level of files accessible via the web. So, if you have a system that uses ‘public_html’ as your web root, you’ll have to modify this set up to have all Drupal core files located within the root of ‘public_html’.

Dreamhost allows you to be more flexible about where to put your web root for each domain. The default is ‘/home/username/domain.tld’, however you can change this. Under ‘specify your web directory’ I list ’/home/username/ceardach.com/production/drupal’.

If editing your httpd-vhosts.conf file directly, your web root is the ‘DocumentRoot’ setting.

Here is a section of my subversion file structure:

- Drupal   - Tags     - drupal-5.0     - drupal-5.2     - ...     - drupal-5.7     - drupal-6.0     - drupal-6.1     - drupal-6.2   - branches     - drupal-5.x     - drupal-6.x - dbscripts   - trunk - ceardach.com   - trunk     - x drupal (external to drupal-5.7 tag)     - x dbscripts (external to dbscripts trunk)     - sites       - all         - modules         - themes       - default         - files - kmgallery.com   - trunk     - x drupal (external to drupal-5.7 tag)     - x dbscripts (external to dbscripts trunk)     - sites       - all         - modules         - themes       - default         - files

Then when I run a checkout of files it will automatically grab all Drupal and dbscripts files along with it. Since dbscripts externals are pointed towards trunk, the sites are automatically updated whenever dbscripts is updated.

One downside is there are some modules that modify Drupal core files. The image module has an optional include file that has to be placed in the ‘/includes’ folder in order to be used. jQuery update module actually modifies a core file. Since these two changes are generally innocuous and used on virtually all of my sites, I just include them within my version of Drupal core in subversion.

However, managing more extensive patches to core may be end up being more complicated to manage. But then again, managing any changes to core is complicated, which is why you shouldn’t do it.

I know there are some to-die-for patches out there, so my recommendation would be to create a branch for tracking a version of core files that contains the patch. When you update your non-patched version of Drupal within subversion, you can then merge in that change commit into the branch that contains your patch. That should help ensure you don’t overwrite the patch’s changes to core, while simultaneously alerting you to any conflicts between the patch and the update to Drupal.

Original Article:

Development Environment for Drupal