Testing Frameworks - an Exploration

Language
English

In this blog post, I walk through an automated testing setup, using Behat, Mink and Selenium.

9th February 2015By jamie

Many months ago, a discussion between my colleague Chris Maiden and I sparked off an idea that we developed into an automated testing framework, which is capable of taking any assertions from user stories, and running them either as unit tests against code, or as functional tests against a staged or development web interface.

There are two main reasons for developing this. The first and simplest is that our clients want it, and we like to help our clients. The second is that it makes sense to have a set of standardised ‘Drupal’ web tests which we can execute in part or full against builds, to ensure that we’re not breaking critical behaviour. A library of standard tests would be beneficial on any current or future projects.

Enter Behat

Behat is a testing framework for PHP, based on work done for Ruby (which resulted in a project called Cucumber). Since there’s a natural link to long, green vegetables, the syntax shared by both behat and cucumber is called Gherkin - no I’m not sure where this came from.

Essentially, Behat exists to make unit tests easy. The gherkin syntax is fairly close to natural language (as scripting syntax goes). An example:

# features/search.feature
Feature: Search
In order to see a word definition
As a website user
I need to be able to search for a word

Scenario: Searching for a page that does exist
Given I am on "/wiki/Main_Page"
When I fill in "search" with "Behavior Driven Development"
And I press "searchButton"
Then I should see "agile software development"

You can probably follow this. The awesome thing about Behat is that it takes this syntax (this is actually valid) and runs tests. The orange bits are keywords. The green bits are variables. Within the scenario block, the bits between keywords and variables actually map to PHP functions in the background, which are stored within context classes.

Alone, Behat would be used to run unit tests, with contexts providing accessor functions for PHP units, which provide assertions to test. As useful as this may be, it’s arguably not always the best way to test a website, which is where Mink comes into play. Mink is effectively an abstraction layer, which lets Behat talk to different functional testing frameworks. It provides a handful of web oriented contexts, but most importantly, it executes gherkin syntax test cases in browser emulators, or live browser sessions, using any framework you’d care to name! In our case, we’ve used Mink’s default settings to plug into gouette (a ‘headless browser’ implementation) and selenium (a framework which allows tests to be run directly through the browser).

So this is all great. We can use gouette for all test instances where we do not require an actual browser. However, some of our functional tests are going to require Javascript to run, and in those cases (at least) we’ll need Selenium.

Watching Selenium in action is a little bizarre. When you run tests, it literally opens up a browser, and starts clicking on things, typing into search boxes, going to URLs you’ve specified. But having seen it run, you begin to wonder how it would ever work on a server, running automated tests post-build. In most cases, our servers are running a linux distribution which doesn’t even have a desktop GUI installed, let alone a browser, but there’s a trick to getting this to work; a little package called Xvfb. It stands for X-windows Virtual Frame Buffer, and gives you a ‘virtual’ desktop GUI in which you can run standard applications, which would otherwise require a desktop. You can’t see it, because it doesn’t actually have a front end, but your desktop applications run in it, and return whatever output they’ve been designed to return.

Setting it up:
Behat, Mink & Selenium

The default, and by far the easiest way to install Behat is to use Composer - which is a tool employed by Symfony (and other PHP projects) to resolve dependencies in a graceful and automated way.

In the root of our project, we’ve created a behat/ subdirectory, and in it created a file called composer.json, which contains the following JSON structure:

{
"require": {
"behat/mink": "*",
"behat/mink-goutte-driver": "*",
"behat/mink-selenium2-driver": "*",
}
}

Having saved this, we run the following commands from the terminal, in our behat/ subdirectory:

$ curl http://getcomposer.org/installer | php
$ php composer.phar install

This will download a file called composer.phar - a compressed PHP application which will download and install all the dependencies, plus the Behat, Mink, Gouette and Selenium packages required for this exercise. Once present, they’ll live in the behat/vendor/ subdirectory.

There’s a little more work to do before we can test this out. First of all we’ll need to initialise a behat test project. From our behat/ subdirectory we run:

$ bin/behat --init

This will create a project structure for us, which will consist of a behat.yml file, and a behat/features/ subdirectory, which is where our tests will live. By default, behat.yml is not set up to autoload all of Mink’s extra classes within feature tests, so this will need to be edited so that it looks like this:

# behat.yml
default:
extensions:
Behat\MinkExtension\Extension:
goutte: ~
selenium2: ~

this tells Behat that any tests not defined as using a user-defined context, will use the MinkContext class by default, which means they’ll know how to perform all the defined common web-related actions. We can test this is the case by running

$ bin/behat -dl

from within the behat/ subdirectory. This should return a list of contexts which Behat/Mink knows about.

Assuming this all works, next we’re going to need some tests! We’ll save the following in a file at behat/features/search.feature:

# features/search.feature
Feature: Search
In order to see a word definition
As a website user
I need to be able to search for a word

Scenario: Searching for a page that does NOT exist
Given I am on "/wiki/Main_Page"
When I fill in "search" with "Glory Driven Development"
And I press "searchButton"
Then I should see "Search results"

The astute among us will notice that the feature defined above is checking a wiki’s search functionality. Wikipedia, to be exact. We need to tell Behat which site it is we’re running tests on, which happens back in the behat.yml file:

# behat.yml
default:
extensions:
Behat\MinkExtension\Extension:
base_url: http://en.wikipedia.org
goutte: ~
selenium2: ~

Once that’s done, we’re ready to run our Behat tests!

As it stands, we can execute these tests by running:

$ bin/behat

in our behat/ subdirectory. By default, Behat will use the gouette driver to run tests, so you won’t see anything spectacular happen - just some test results in your terminal window. Next, we need Selenium.

Testing Selenium Locally

First off, to test using Selenium, we’re going to need a Selenium server running. It can be downloaded as a .jar file from http://seleniumhq.org/download/ and running it is a simple matter of getting into whichever directory it’s saved to, and running:

$ java -jar selenium-server-*.jar &

Obviously, we need a JRE installed, and we could replaced the ‘*’ with the specific file name or version number we downloaded.

Once we have Selenium running in the background (hence the &), we need to modify our feature file like so:

@javascript

The ‘@javascript’ tag above the Scenario tells Mink to use a driver which supports, predictably, javascript (and therefore will load the test in an actual browser, rather than an emulation of a browser).

If we now run

$ bin/behat

from our behat/ subdirectory, we should see Firefox open up, navigate to the Wikipedia site, and start running our tests. When it’s finished, it will close again, and our test results will be displayed in the terminal.

Xvfb, & Testing Selenium on a headless server

So, having got all the following working, we need to take it a step further. Supposing we want our test framework to run on a server machine - one that’s hosted at Linode, or Rackspace, and which we don’t want to install Gnome, or KDE or even Windows. We could just use the Gouette driver to execute tests, but that doesn’t allow us to test any javascript-rich functionality. We need a way of emulating the desktop environment, and running a browser.

This is where a package called Xvfb comes in. We can find it in most package repositories - my favourite flavour of Linux is usually Debian or Ubuntu, so we’d install it with:

$ sudo apt-get install xvfb

And it can be run with:

$ Xvfb :99 -ac &

This means to run Xvfb on a display we’ve numbered 99, with no user access control, and send it to the background.

We can now start up Selenium, and run our tests on this server (assuming we’ve deployed our tests on this server), but we’ll need to tell Selenium which display to open browser instances on:

$ export DISPLAY=:99
$ java -jar selenium-server-*.jar &

and now we’re good to go. As before, we go to our behat/ subdirectory, and run:

$ bin/behat

We’ll see the same output as we did when running tests in Selenium locally, and we should also get output from Selenium and Xvfb, since they’re running as background tasks in the current shell.

Oh no, my server needs a different setup to my local machine/vm!

When we first committed our Behat framework to a project, pushed it to a stage server, and tried to execute our tests, we found that there were dependency problems. This is largely because when Behat is installed, it uses the Composer framework to check which libraries are needed, downloads them automatically, and includes them from a vendor/ subdirectory which is created as part of the install task.

Since the vendor/ subdir is the only place in which these dependencies are resolved, our solution was to install Behat/Mink locally to the server via composer, somewhere safely away from the project. Then when code was pushed up to our stage server, it’s a simple task to remove the vendor/ subdir from the project, and symlink in one which we’re confident contains all the dependencies for the current server.

Another way we could tackle this problem could be to install Behat/Mink/Selenium/Xvfb on our server as before, and have Behat’s features/ subdir version controlled, and managed by a CI setup, such as Jenkins. That way, our tests would be separate from our project, and could be run as needed without worrying about dependencies at all.

Running Selenium/Xvfb as a service

There are a few ways of having Selenium and Xvfb running in the background on our server, without having to run them explicitly from the terminal each time we need them. One way we explored was to have them installed as ‘service’ scripts, from /etc/init.d/ but the simplest and easiest way we opted for was simply to place the following in /etc/rc.local, which executes commands when the machine starts:

/usr/bin/Xvfb -ac :99 &
export DISPLAY=:99
java -jar /usr/local/bin/selenium-server-standalone-2.25.0.jar &
exit 0

This is enough to have the necessary programs running when we need them, which is especially useful if we’re integrating our test framework with a Continuous Integration framework, such as Jenkins. Which is exactly what we intend to do ;-)

Jenkins, Fabric.py and Continuous Integration

The idea behind this is to have tests run whenever a changeset is pushed to a git branch. As we get out into the realms of CI setups, there are fewer and fewer absolutes, because each CI setup is different depending on who created it, and where and why it’s been set up.

In our case, we’ve added a step to our fabric file, which looks like:

# Run behat tests, if present
def run_behat_tests(repo, branch, build):
if os.path.isdir(cwd + '/behat') and env.host == 'staging_host_name.codeenigma.com':
print "===> Re-linking vendor directory"
run("cd /var/www/%s_%s_%s/behat && rm -rf vendor && ln -s /var/www/shared/behat/vendor" % (repo,branch,build))
print "===> Running behat tests..."
run("export DISPLAY=:99 && cd /var/www/%s_%s_%s/behat && bin/behat" % (repo,branch,build)

All this is doing, is checking to see if a behat/ subdirectory exists, and that the hostname we’re deploying to matches our expectations for the for the environment on which we’d like to run tests, and then if these criteria are met, we remove and then symlink the behat/vendor/ subdirectory to one we know is good for this server, and then run the test suite (having exported the display variable).

Ultimately, a continuous integration setup is going to vary depending on the individual circumstances involved, but building a ‘test’ step in is pretty simple. Behat returns FALSE if tests pass, and an error message if not, so it’s very simple to check for.

It’s also possible to run Behat with command line options to output the results to HTML, so this can be combined with a CI framework to just email the results of any tests, and pass the build regardless - again, it’s down to the individual needs of the project.

(Main banner image: dummies by Greg Westfall)

An Introduction to Test Driven DevelopmentBlog

Getting started with Test Driven Development - Choosing a Test HarnessBlog

How do Code Enigma provide continuous integration?FAQ

Meaningful commit messagesBlog

Original Article:

Testing Frameworks - an Exploration