Report from the University of Minnesota Drupal Usability Testing

From May 17 - 19, 2011, in advance of DrupalCamp Twin Cities, several Drupal community members met at the University of Minnesota usability lab in Minneapolis to perform a round of formal usability testing on Drupal 7. This is the fourth major usability testing for Drupal, and the first targeting the new Drupal 7 release.

People who were familiar with building websites (but not with Drupal) were observed while they worked through a number of site building tasks. This report contains a summary of the results.

Summary

The good news is that most of the changes that were put in Drupal 7 tested well. Compared to Drupal 6, Drupal 7 no longer confuses new users with basic conceptual hurdles like where the front-end vs. back-end of their site is and how to create an "About us" page, and for the most part the administrative interface is clear.

The bad news is that now that some of these basics have been dealt with, we've uncovered a whole new layer of challenges for first-time Drupal site builders, some of which were pretty surprising. Finding modules, creating and placing blocks, and creating content types were difficult tasks for participants to understand, and it's these site builder tools we'll want to improve for Drupal 8.

Who did we test, and how did we test them?

Eight participants (a typical number for this kind of study) evaluated Drupal by working through a series of real world tasks over a 75 minute period, "thinking aloud" while the team observed and took notes from behind a one-way mirror. The participants could phone a "help desk" (staffed by a member of the team) if they ever got stuck and needed a nudge in the right direction, and a number of open-ended questions were asked throughout the test session.

All eight of our participants are people directly in Drupal's target audience. They are site builders, already using tools such as Dreamweaver and WordPress. They know HTML and understand how the web works. All but one had no prior experience in Drupal.

What did we ask them to do?

First, we gave participants a couple of minutes to click around the interface and provide their initial impressions. Then, we provided them with a series of tasks (see the scenarios covering basic things site builders do from day-to-day, including:

Content creation
Appearance settings
Block management
Installing a module
Creating a new content type

Test Results

In previous usability testing against Drupal 6 and early Drupal 7, we had intended to test things such as CCK, users and permissions, taxonomy, and so on. What we ended up testing instead was how horrifically confused new users were performing even basic tasks within Drupal 6's administrative interface: the separation of front end/backend, the overwhelming number of options at /admin, confusion between Page and Story, and more all stopped people from using Drupal successfully.

Based on this testing, we made a number of changes to Drupal 7 to try to combat these conceptual problems. And in short, all of the major usability problems found in Drupal 6 and attacked in Drupal 7 appear to be either vastly improved or non-existent in Drupal 7. Hooray! :D

What tested well?

First impressions: After some initial exploration (around 2 minutes), participants had a favorable response to Drupal 7 and thought it was:

"very simple"
"clearly laid out"
"there is a lot you can do"
"not very cluttered"

Creating content: All participants—but one—were able to add content (using different links from the homepage) without difficulty. Participants thought it was "pretty simple", "straightforward" and "easy to find 'add content'".
Basic Page vs. Article: Participants understood the difference between "Basic Page" and "Article". All the participants who tried to add an "About us" page to the main menu chose "Basic Page".
Changing the color of their site: Participants instantly knew to go to "Appearance" to change the color of their site. Although some participants had minor issues finding the theme settings page, they were eventually able to complete the task. Participants also liked how of color changes were previewed live on the affected area itself.
Toolbar: Participants found the toolbar quickly and had a fair understanding of what it did.
Toggle between the administrative overlay and site: All but one participant had a clear understanding of how to toggle between the overlay and the site. Participants understood when they were performing administrative activities and frequently used the overlay's "close" button to return to their previous location or before preparing to start a new task.

What tested poorly?

Now that people can actually navigate and do basic content creation tasks without much difficulty, they encounter other parts of Drupal's administration that are still utterly baffling. This section covers the major issues, though there are many more, some of them quite easy to fix!

Drupal terminology is obscure.
Terms like "Block" and "Module" and "Content types" and "Triptych" either have no meaning, or a preconceived meaning that is different from what Drupal uses these words for. For example, some think of "module" as a block.
The distinction between "sidebar" content and content made absolutely no sense. Content is content.
When asked to create a sidebar block, every single participant went for the "Add content" link. Unsure which option to choose, they went for either "Article" or "Basic page" and scoured the vertical tabs options for a way to add it into the sidebar.
The lack of a visually obvious way to place content on the page was a huge detriment
People really craved some sort of "edit in place" mode as a visual way of controlling their site layout. They tried to click on region names in the block demonstration page to add content there. They tried to take a node that was outlined with the contextual links border and drag it to the sidebar. They tried to click and drag to resize sidebars. The lack of this functionality made Drupal appear extremely clunky.
Lack of realistic previews makes building sites in Drupal extremely challenging
The preview as provided by Color module is what participants really were craving to see absolutely everywhere.
Menus have no preview either, only text descriptions that give no visual clue to where on the page they will appear. Content can be previewed, but this preview is shown in the administration theme and overlay, not in the context of the actual user-facing site. This was frustrating and confusing.
While Drupal takes a "content-first" approach to site building, many people take a structure- and/or appearance-first approach.
When looking at a mockup of the website they were trying to build, the natural inclination for several participants was to start setting up the navigation structure first. They ran into issues when doing this, because they were required to enter something for the "Path" field of the menu item they were creating, and they had no idea what that could be.
These Drupal evaluators did not realize you can extend it, and so Drupal is seen as limited in functionality.
If users found their way to the modules page at all, they did not see that they could add new options there.
Given that the entire point of using Drupal is that you can easily extend it with modules, making this fact more discoverable should be a primary objective of Drupal 8.
The workflow of finding and adding a new module from Drupal.org is overwhelming.
Once a help desk call pointed them at the "Install new module" link, new problems began. Being taken off-site to Drupal.org was extremely jarring, as was the (apparent) requirement to download a module to the desktop and then upload it to the server to install it. People suggested an in-app browser for modules from Drupal.org. Additionally, the filtering options on drupal.org for modules are overwhelming. The search box, arguably the most valuable tool for when you have a general idea of what you're looking for, is located far down the list and was missed by most participants. Instead, they honed in on the "Categories" drop-down, which was completely useless to them.
People were unable to make the connection between a module's version number and modules listed on Drupal.org
They generally just picked the first thing in the list, which happened to be 7.x for Webform, but wasn't for "Announcements", which another participant searched for. (Another issue is that once people know modules exist, they want to use them for absolutely everything, including things that could be accomplished through blocks and content types.)

Overall Results

After 75 minutes of using Drupal 7, participants were asked about their overall expressions. Participants commented:

"This was just hard. I struggled with it. I will have my sister help me."
"If you know what are you doing, it's okay. But for the first time user, it is a little shaky."
"A little confusing at first ... but I think you need to toy around with it ... get familiar with it."
"Not as user friendly as I thought."

When the participants were asked to select five words they associated with their Drupal 7 experience, 60% of their comments were positive. The most prominent word was "Customizable". (See the complete results.)

Participants were also asked to rate their experience on a 5 point Likert scale in terms of ease of use and value. Half of the participants thought Drupal was "Difficult to use", and all the participants thought it was "Completely valuable" or "Somewhat valuable". (See the complete results for ease of use and value.)

Finally, it is important to note that none of the participants were able to make it through their session without needing at least some assistance from the "help desk".

Next steps

We found over 100 issues during this testing round, some of them major, and many more minor ones. These are captured in raw form in the "Issues Analysis Matrix" spreadsheet, and in the issue queue under the "UMN 2011" tag. We will discuss, design and code changes for these, involving the larger community on what it means to fix them, and how. You are most welcome to join the effort!

Keep tabs on high-impact issues at the Usability Community Initiatives page.
Participate in larger discussions in the Usability group.
Provide and review patches for issues with the "UMN 2011" tag.
Help us create Drupal.org issues for the remainder of the issues found in testing. (Ping someone in #drupal-usability for edit access)
Help with uploading and splicing videos of participants' experience to illustrate issues to developers in the issue queue. (Ping someone in #drupal-usability for more info)
Chat with the team in #drupal-usability and #drupal-contribute on IRC.

We also encourage you to help with further usability testing, both formal and informal. What we tested at University of Minnesota only scratches the surface of what Drupal core can do (for example, we didn't cover Taxonomy, Search, adding themes from Drupal.org, etc.). We need further testing to verify if our changes are actual improvements, and there are many contributed modules out there that would benefit from usability testing as well.

Join us and help make Drupal 7 and 8 a joy to use!