The Drupal study #3 team came up with some pretty interesting tricks to use time efficiently and capture data during the usability study. I geek out on tools, and how we make or hack them on the fly (and whether they are physical or process), and want to take a moment to document what everyone did before that walrus slides off the iceberg into cold sea…

First off, context. Our usability study was fairly informal – we had a range of participants, and observed as they worked through a basic “build a small conference website” scenario. However, we only had four days to analyze the data from the eleven sessions and put together a presentation of the results. Bojhan and I spent the few weeks ahead of time figuring out the test plans and coming up with a general idea of how the week would work. In our minds it went: task people with note-taking during the sessions / do a review of observed issues after each session and capture the issues on post-it notes organized by task / review issues and video after sessions / make it all make as much sense in a presentation asap.

Once we all got in the lab together, new ideas started to sprout up.

When I’m moderating a session, I like to be left alone. If people start whispering their thoughts or questions while I’m watching or thinking, I easily lose my train of thought, and get pretty grumpy about it. Bojhan came up with what turned out to be a great idea for me – creating a Twitter account and having people in the room direct message me throughout the session. I had it on refresh every second so I could get messages right away, and just had my laptop out in front of me by the eyetracking-viewer. If I needed to clarify something with whoever asked the question, I could turn around and ask, but I could do it on my own time.

Bojhan wanted to avoid asking questions in an IRC chat room to keep notetakers from getting distracted. But I believe it was Nat suggested using IRC to take notes – which captured a timestamp everytime something was noted. Because the time on the computer is recorded during the session, this prooved to be a great help later when I went back to pull video clips.

Our time constraints pretty quickly brought up the problem of how to get all of the issues captured and available to the community. In a room of madly skilled web developers, a drupalusability website was soon up and running. It started as a place to write out the usability issues, and label them by task and participant.

Later categories were used to group the different issues (such as labeling + help, conceptual and findablity). The site can be sorted on any of these. It was a great help for me in pulling video clips as well – I could choose a category, see all the related issues, choose one that’s relevant and see all of the participants who had that specific problem (as well as see where it occurred thanks to the notes with time stamps). The site also includes video from all of the sessions, so other community members can participate in analysis.

The website is still a work in progress, and is likely to house many user research, design and evaluation work in the future. The Drupal usability group is also talking about creating an install profile to make available to the community.