One little practical issue I found myself thinking about during the Drupal study was accommodating different styles of working while evaluating a web app. Something we wanted all of our participants in the study to take a look at was the Menus page. During testing, some participants went (at least somewhat) immediately to blocks to add a link on the left hand side. There goes talking to them about menus.

This is totally appropriate, and from what I gather, a better option than what we wanted from the test plan (adding a link in the Navigation Menu). But what do you do when half-way through your study you still want people to take a look at that Menus page? There are plenty of ways, sure – pulling up the page and just asking to start. Or asking why they didn’t use a menu. But as moderator, it got me thinking about the test plan in general.

The different ways of approaching a task teach us a lot about how people have learned to use software, and seeing how they would naturally do something is the most valuable. At the same time, I felt at the time it was important to make sure participants worked with and reacted to the Drupal menu set-up.

So with that loosely in mind, here’s some of my post-Drupal study #3 brain dump:

1. Clear goals. It’s difficult at the moment to define goals when evaluating Drupal. Without a clear sense of the target audience(s) and some of the actual key moments that these users face early on with Drupal, it’s tough to determine the best way to go about a test plan. I think our more exploratory studies have taught us quite a lot, but from here on evaluating Drupal would be better off more specific and goal oriented.

Example, this time around we wanted to test the installer with (pretty great) new password checker, because install is the obvious first hurdle for all users. This was a manageable goal, and a good one. Now, what comes next? Some of our participants were uncomfortable posting any content before setting up and organizing their site – makes total sense, it’s the more typical workflow (from what I’ve seen in interviews). So who’s the target audience, and what are their common tasks and workflow when getting started with Drupal? We need to figure this out more clearly to better evaluate how Drupal works. Which leads to,

2. More user research. Sure it’s a lot of work, but all of it, including research methods pulled from marketing and stats from drupal.org, will help set clearer goals in evaluation and design. Good design won’t happen out of context.

3. A more flexible test plan with a greater emphasis on seeing how people work. Or perhaps just an easier way to categorize some of the helpful information about users we have, like ‘I wouldn’t put in content first, I’d go configure the site’.

post-it issue