Automated testing 2.1 deployed - contributed projects

After a lot of work, waiting, staging, and such I am proud to announce the addition of contributed projects to the automated testing system. Contributed projects may now take advantage of the same system that Drupal core developers have been using for over a year with great success. The deployment comes quickly after the recent 2.0 launch in late November of 2009. In addition to adding support for contributed project testing a number of other features have been added, most notably:

Coder and Coder Tough Love review support.
General e-mail notifications - the devlist mailing list will get an e-mail when Drupal core breaks.
A number of UI/workflow improvments on drupal.org.
Grouping of reviews by type or plugin to make room for a cleaner multiple database testing workflow.
Additional administrative tools for qa.drupal.org and testing clients.
Views RSS feed plugin for aggregation of test events and results.

For an example of the test results, please take a look at the current Drupal 7 (HEAD) results or one of our beta contributed project results, such as poormanscron.

What this means for contrib project maintainers

If you choose to enable automated testing on your project(s), you may take advantage of the system in a variety of ways. For those project maintainers that simply make commits without issues, the automated testing system will queue testing of the related branch after each commit. The results will be available on qa.drupal.org; in the future, they will be displayed on project pages. If maintainers choose to use a patch-based workflow and/or receive patches for their project(s), the patches will also be automatically tested.

Since the automated testing system checks for common problems like a patch failing to apply, PHP syntax errors, Drupal installation failure, or test failures, it makes it much quicker to weed out the bad patches from the good ones which, in turn, saves maintainers time. In addition, having the automated testing system helps ensure that any tests written for a project are run and thus increases the value of tests. By increasing the value of tests, we hope this will motivate developers to write tests for their projects.

Call to action

With the upcoming Drupal 7 release and the subsequent porting of projects to the new API, there is a great opportunity to write tests for modules as they are being ported to ensure that they remain functional. Since the automated testing system also supports testing in Drupal 6, tests can be written before a project is ported and then the same tests can be run in Drupal 7 until they pass, signifying that all functionality works again.

In the future, there are a number of enhancements planned for the system which will increase the benefits of writing tests. We hope the community as a whole sees the value in writing tests for contributed projects and takes advantage of the automated testing system to improve the overall quality of the Drupal codebase.

Beta stage

To ensure that everything is working well, the system will be restricted to selected contrib projects. During this beta stage, we will ensure that everything functions properly and that the test client network can handle the additional load. After making any necessary adjustments, we will open up the option to all contrib project maintainers. If you would like to have your project included in the beta stage, please comment on #689990: Contrib projects to be included in beta stage of automated testing for modules.

System limitations

In order to test all contrib projects, the system must determine the dependencies of each project in order to checkout that code when testing. Using the Drupal 6.x style .info files, this cannot be done with a 100% accuracy since no version information is included. As such, the current system will attempt to choose the "best release" for each dependency. The system will use the most recent development release with an accompanying stable release. This limiation does not exist with the Drupal 7.x API as the .info files can specify version restrictions, such as >=2.0.

Due to this limitation, certain projects with dependencies may not be tested properly for the time being.

To provide a common example of a dependency that will cause the system to fail, let's look at the current state of the Views project. The majority of projects will depend on the 2.xVviews API, but since there is now a stable views 3.x release (the alpha), the 3.x development release will be used for testing purposes. Consequently, any modules that depends on Views cannot be tested until this has been resolved.

The solution will most likely be a format for specifying version limitations as in Drupal 7 that does not break Drupal 6 compatibility. Once we have decided on the exact format and implemented the version checking into the dependency resolving algorithm, this problem should be fixed.

Donating test clients

Donating a test client has never been easier thanks to the AMI provided by Chapter Three. Simply deploy the AMI, request an account on qa.drupal.org, add the generated API key to the test client, and you're set. The system will automatically be tested to ensure everything is functioning properly, request tests from the testing master (qa.drupal.org), and report the results of each test.

With the addition of contributed projects, we will most likely require a few extra testing clients to keep up with the increased demand. Any donations of time or machines are appreciated. Please use the contact form on qa.drupal.org if you have any questions or concerns.