Unit testing in Drupal using the standard SimpleTest approach has long been one of my pain points with Drupal apps. The main obstacle was setting up a realistic test "sandbox": The SimpleTest module builds a virtual site with a temporary database (within the existing database), from scratch, for every test suite. To accurately test the complex interactions of a real application, you need dozens of modules enabled in the sandbox, and installing all their database schemas takes a long time. If your site's components are exported to Features, the tests gain another level of complexity. You could have the test turn on every module that's enabled on the real site, but then each suite takes 10 minutes to run. And that still isn't enough; you also need settings from the variables table, content types real nodes and users, etc.

So until recently, it came down to the choice: make simple but unrealistic sandboxes that tested minutia but not the big-picture interactions; or build massive sandboxes for each test that made the testing workflow impossible. After weeks of trying to get a SimpleTest environment working on a Drupal 6 application with a lot of custom code, and dozens of hours debugging the tests or the sandbox setups rather than building new functionality, I couldn't justify the time investment, and shelved the whole effort.

Then Moshe Weizman pointed me to his alternate upal project, which aims to bring the PHPUnit testing framework to Drupal, with backwards compatibility for SimpleTest assertions, but not the baggage of SimpleTest's Drupal implementation. Moshe recently introduced upal as a proposed testing framework for Drupal 8, especially for core. Separately, a few weeks ago, I started using upal for a different purpose: as a unit testing framework for custom applications in Drupal 6.

I forked the Github repo, started a backport to D6 (copying from SimpleTest-6 where upal was identical to SimpleTest-7), and fixed some of the holes. More importantly, I'm taking a very different approach to the testing sandbox: I've set up an entirely separate test site, copied wholesale from the dev site (which itself is copied from the production site). This means:

• I can visually check the test sandbox at any time, because it runs as a virtualhost just like the dev site.
• All the modules, settings, users, and content are in place for each test, and don't need to be created or torn down.
• Rebuilding the sandbox is a single operation (with shell scripts to sync MySql, MongoDB, and files, manually triggered in Jenkins)
• Cleanup of test-created objects occurs (if desired) on a piecemeal basis in tearDown() - drupalCreateNode() (modified) and drupalVariableSet() (added) optionally undo their changes when the test ends.
• setUp() is not needed for most tests at all.
• dumpContentsToFile() (added) replicates SimpleTest's ability to save curl'd files, but on a piecemeal basis in the test code.
• Tests run fast, and accurately reflect the entirety of the site with all its actual interactions.
• Tests are run by the Jenkins continuous-integration tool and the results are visible in Jenkins using the JUnit xml format.

How to set it up (with Jenkins, aka Hudson)

(Note: the following are not comprehensive instructions, and assume familiarity with shell scripting and an existing installation of Jenkins.)

1. Install upal from Moshe's repo (D7) or mine (D6). (Some of the details below I added recently, and apply only to the D6 fork.)
2. Install PHPUnit. The pear approach is easiest.
3. Upgrade drush: the notes say, "You currently need 'master' branch of drush after 2011.07.21. Drush 4.6 will be OK - http://drupal.org/node/1105514" - this seems to correspond to the HEAD of the 7.x-4.x branch in the Drush repository.
4. Set up a webroot, database, virtualhost, DNS, etc for your test sandbox, and any scripts you need to build/sync it.
5. Configure phpunit.xml. Start with upal's readme, then (D6/fork only) add DUMP_DIR (if wanted), and if HTTP authentication to the test site is needed, UPAL_HTTP_USER and UPAL_HTTP_PASS. In my version I've split the DrupalTestCase class to its own file, and renamed drupal_test_case.php to upal.php, so rename the "bootstrap" parameter accordingly.
   ** (note: the upal notes say it must run at a URL ending in /upal - this is no longer necessary with this approach.)
6. PHPUnit expects the files to be named .php rather than .test - however if you explicitly call an individual .test file (rather than traversing a directory, the approach I took), it might work. You can also remove the getInfo() functions from your SimpleTests, as they don't do anything anymore.
7. If Jenkins is on a different server than the test site (as in my case), make sure Jenkins can SSH over.
8. To use dumpContentsToFile() or the XML results, you'll want a dump directory (set in phpunit.xml), and your test script should wipe the directory before each run, and rsync the files to the build workspace afterwards.
9. To convert PHPUnit's JUnit output to the format Jenkins understands, you'll need the xUnit plugin for Jenkins. Then point the Jenkins job to read the XML file (after rsync'ing if running remotely). [Note: the last 3 steps have to be done with SimpleTest and Jenkins too.]
10. Code any wrapper scripts around the above steps as needed.
11. Write some tests! (Consult the PHPUnit documentation.)
12. Run the tests!

Some issues I ran into (which you might also run into)

1. PHPUnit, unlike SimpleTest, stops a test function after the first failure. This isn't a bug, it's expected behavior, even with --stop-on-failure disabled. I'd prefer it the other way, but that's how it is.
2. Make sure your test site - like any dev site - does not send any outbound mail to customers, run unnecessary feed imports, or otherwise perform operations not meant for a non-production site.
3. In my case, Jenkins takes 15 minutes to restart (after installing xUnit for example). I don't know why, but keep an eye on the Jenkins log if it's taking you a while too.
4. Also in my case, Jenkins runs behind an Apache reverse-proxy; in that case when Jenkins restarts, it's usually necessary to restart Apache, or else it gets stuck thinking the proxy endpoint is down.
5. I ran into a bug with Jenkins stopping its shell script commands arbitrarily before the end. I worked around it by moving the whole job to a shell script on the Jenkins server (which in turn delegates to a script on the test/dev server).

There is a pending pull request to pull some of the fixes and changes I made back into the original repo. In the pull request I've tried to separate what are merely fixes from what goes with the different test-site approach I've taken, but it's still a tricky merge. Feel free to help there, or make your own fork with a separate test site for D7.

I now have a working test environment with PHPUnit and upal, with all of the tests I wrote months ago working again (minus their enormous setUp() functions), and I've started writing tests for new code going forward. Success!

(If you are looking for a professional implementation of any of the above, please contact me.)

Recent related post: Making sense of Varnish caching rules