19 Aug

Stuart Clark

At Realityloop I've been involved with some extremely complex site builds, varying from Commerce driven paywalls to fully fledged data reporting web applications. Yet, the hardest thing I've ever had to do is successfully sell automated testing to a client.

And the reason for that is relatively simple; Tests are hard to write, therefore time consuming and expensive.

However this is a misconception, while certain types of tests can be hard to write, any automated tests will recoup the costs in a short amount of time, and as such, you can’t afford not to have tests.

In part one of this post I will be explaining exactly why you need tests. In part two (due in three weeks) I will cover how to get started and implement your automated tests.

What are tests?

A test is exactly what it sounds like; a method to ensure that your project behaves as is expected.

Any steps taken by an individual to ensure the project is behaving correctly are considered to be manual tests, and while they are essential, they are time consuming.

For an example of what a test may be, let's look at the following scenario:

Fictitious Inc want a contact form on their website that will send emails to specific recipients based on the type of enquiry.

They need three enquiry types:

1. Support to support@fictitious.inc

2. Sales to sales@fictitious.inc

3. General enquiry to info@fictitious.inc

Along with this requirement, they also wish to have a personalised auto-response per enquiry type.

Once this the above functionality has been created, all parties, at some point, will need to ensure that it does indeed work as expected.

This process is your manual test, which would look something like this:

Jarkko navigates to his Fictitious Inc. development site, clicks on the Contact menu item, selects the Support enquiry type, fills out some dummy data and submits the form.

He then ensures that the email was sent to the correct email address with the supplied data, and ensures that received the correct auto-response with the correct personalisations given the supplied data.

Lastly, he repeats the process with the other two enquiry types, ensuring that he supplies different data to his previous test(s).

That’s a test, and a relatively simple test, but already it's evident that it is time consuming. Throw in more complexity to the requirements and the time increases exponentially.

Any future work on this project, regardless of the area of work, should ensure that these same tests, as well as any other tests in the site, are run again. Otherwise you run the risk of accidentally breaking a core feature of the project.

As such, manual testing becomes excessively time consuming, and often a step that is skipped by all involved.

Automated testing is the answer.

What are automated tests?

There are various types of automated testing, but for the sake of this post I will be concentrating on "integration tests".

An automated test is, again, exactly what it sounds like; an automated method to ensure that your project behaves as is expected.

Take the previous scenario, currently each individual involved in the project (developer, tester, project manager, client, etc) could be spending anywhere between 2 and 10 minutes to manually test that small subset of the project functionality every time any changes are made to the project.

Automated, the same test should take no more than a 30 seconds in total, and can be run along with all other automated tests in one quick hit.

Manual testing vs Automated testing

In the above video you can see the Automated test running side-by-side with the Manual test.

The Automated tests are not only testing the same scenario as my Manual tests, but also installing a fresh copy of Drupal, enabling all required modules, flushing caches, reverting features and running a few additional checks that I didn't do in the Manual test.

For those of you who do not wish to watch the video, the results are as such:

Manual testing - 3 minutes 19 seconds

Automated testing - 27 seconds

However, there is still a time investment involved to create the tests. In this case it took me roughly 60 minutes to write the tests, in which time the manual tests could have been run about 18 times, assuming of course your manual tester doesn't decrease efficiency after running each test.

However...

Continuous Integration

With the above simpletest, the responsibility of running the tests is still in the hands of a human, and as such you will only get your test results if someone bothers to run the tests.

Continuous integration relieves your human workers of that responsibility, instead automating the running of the automated tests.

Travis CI is one such service which is relatively well loved by the open source community due to the direct integration with Github.

Along with the benefit of running automated tests on every commit (or tag, if that is how you wish to configure it), Travis can run multiple testing jobs concurrently, allowing you to ensure you project works on a variety of different environments.

Travis CI jobs do take longer than running a Simpletest (based on my above video), which is due to Travis not only doing all the additional things that Simpletest is doing but also having to setup a complete environment prior to installing Drupal. However, as you can run potentially upwards of 10 concurrent jobs (on a Premium plan) this time loss is quickly negated.

Assuming you are on a free plan, and the current load on the server allows for 4 concurrent jobs taking a maximum of 1 minute and 20 seconds (the average from my own tests), you recoup the cost of the test development in only 4 commits.

Adding Travis integration to your Drupal project is relatively simple thanks to the Drupal Travis Integration project. It’s as simple as adding and configuring single file to your Github repository. However I’ll go into more detail on this in part 2 of this post.

drupaldrupal planet