Cheap Drupal Website Usability Testing: Practical Lessons and Experiments via Amazon’s Mechanical Turk

You need to know if your site makes sense to your users. You don’t know what your users don’t know, don’t understand, etc.. You need feedback. This is first requirement of any sensible approach to usability and usability testing.

In this post, I’m going to present a personal experiment and reflections on “outsourcing” my usability testing via Amazon’s Mechanical Turk program.

Usability improvements have become an obcession in the Drupal sphere (and for me personally!). There are quite a few recent projects, research studies, speeches and articles about improving Drupal usability. So while reading some books, adhering to usability checklists and following some blogs “on usability” will help, eventually it all comes down to testing.

You need to know what real people think.

So are you doing user testing?

If you are like me, the major of user testing we do is via client testing: do our clients understand how to use the site? Or on personal sites, you might ask your friends and family to check on something. In either case, we are not necessarily getting fresh eyes on the case.

As developers and builders, we become pretty biased about our sites and how they work. We know how they work since we built them. Rarely does that reflect how real folks use something we built. If you are the one doing the development, design and testing, it’s time to (as they say in the startup world) step outside and get some real feedback.

For example, a recent discussion on client project was stagnating in what we should fix and what wasn’t working. We were equally biased in what we thought worked and didn’t. So I proposed we get some general user feedback via a nifty and cheap little service called Feedback Army.

Basically, Feedback Army provides a simple integration with Amazon’s Mechnical Turk system where you present a site link and provide some questions for feedback. After paying $20 (for ten real user responses) via PayPal, the human intelligence task is launched and you just have to wait for the feedback from users.

It took about 2 hours to get about 1700 or so words of feedback from 11 folks (I rejected one response since it showed no effort). The average user spent around 8-16 minutes reviewing the site and answering 5-6 questions. Overall, I was very impressed by the feedback and level of clarity these responses provided. Most of the questions I asked were qualitative questions (what, where, how, etc.) making yes/no responses impossible. I also asked a few open-ended confirmation questions to check if we were conveying certain basic project information clear enough for normal folks to understand (confession: we weren’t!).

Overall, I was happy with the result, and it advanced the client discussion towards real user issues and not our assumptions of the issues.

Since it’s a client project, I didn’t feel like it was appropriate to share this data publicly. Instead I decided I should “eat my own dog food” and do a user feedback test on my own Drupal site: http://int3c.com.

User Feedback Test for Drupal site Int3c.com (Dec 8, 2012)

A bit of a background: I do mostly Drupal development and much less work on the design side. The original site I had before this current one was quite ugly and unprofessional, though it didn’t seem to cause major problems in finding new clients. Overall, I’m more dev than designer. Evenually though ugly is ugly and so, for my site, I hired a designer to do a full branding, logo and initial homepage for the site. I think he did a really awesome job to freshen and modernize the look as well as draw some international connections and themes. Here are some thoughts on the redesign. Afterwards, I then translated it to html/css and later to an Omega responsive subtheme. Since that first version, I’ve made some tweaks and changes but more or less it hasn’t changed. Generally, I use the site to blog about Drupal and post about recent modules and sites I build. I hadn’t done any user testing before, besides asking a few friends and collegues for their initial impression (they’re my friends so, of course, they liked it!).

For this user feedback test, I used FeedbackArmy again. I looked at Amazon’s Mechanical Turk and the Drupal integration modules, but I didn’t really have time to figure out how to do this directly. Since time is money, spending 2 hours or more to set up something custom is more time-expensive than paying 20 bucks to get it done via something that had already proven that it worked. (Disclaimer: I’d love to see a Drupal-based solution that could do this for cheaper and with some more flexibility.)

I’m not a usability expert, just a curious guy wanting to get some general feedback on what works and doesn’t. Here was my posting:

Hello, I’m collecting some feedback on my website’s new design. Your response will help me know if this design and layout is giving the right messages and hopefully get me more customers.

1. What do you think the purpose of this site is? (ie. selling, informing, entertainment, etc)
2. Who do you think the intended audience is?
3. How did you find the layout of the site? Clear? Understandable? Readable? Etc?
4. What aspect of the site confused you?
5. What would encourage you to return to this site in the future?
6. If you could change something on the site, whether it is major or minor, what would be at the top of the TODO list?

For those readers checking this post later (after I’ve implemented some of these changes!), here are a few screenshots of the site circa Dec 8, 2012: Homepage, Blog Listings, and Portfolio Page.

I was quite sure some folks would have some critical things to share. Frankly I was paying people to tell me their thoughts. I’d of course love to hear only positive feedback but that’s unrealistic. You got to take the feedback as it comes (positive, negative, neutral, etc.), draw your conclusions, and move forward with actionables for improvements.

Anyways, here is a pdf of the full results from my website feedback.

Just the Facts, Jack.

So like the first example, this task had 10 responses in about 2-3 hours. There were a few more “not really attentive” reviewers (3, in fact) than the previous test. I’m guessing this is because the site has less holes and problems, so some people had less to say.

Here are some of the positive points from the reviewers:

• “The layout of this site is very professional. It’s very easy to understand.”
• “I found the web site easy to navigate, and logically laid out.”

Here are some quotes of the negatives:

• “The twitter section seems odd on the website as it doesn’t strike me as a company or entity I would follow on Twitter. “
• “I would not keep the two-column alignment on every page, and especially not on the front page. I think it makes the website look slightly cluttered.”
• “The layout on the site feels a little dated and uninspired, it could be a little more sleek.”
• “Under portfolio I don’t like the main topic pointing out Drupal. I came to see your portfolio of sites – not a page describing your ‘Technology’. “
• “What is INT3C and what is Drupal? The name of the site is INT3C but all the reviews and testimonials are about Drupal.”

Here are some other miscellaneous points:

• “I was a little confused about all the references to “Drupal” and what the relationship is between INT3CH and Drupal. “
• “I see marks email on each page but I would make it more prominent by including it as normal black text someplace where I’m not confused by the passport stamp concept. Contact us page should have phone and email at top – above form. I want to contact you now – not wait for a response to a web form..”
• “The only aspect of this site that I found confusing is some of the tools that they use. Some examples of this is Drupal,node.js, and ruby on rails. I have never heard of such things. I think of myself as tech savey so I can not believe I have never heard of these things.”

Some Analysis and Summary of Usability Feedback

We tend to be pretty biased even when looking at the data, so I advice you to check with the data yourself. Here are some of the points that stuck out in my reading:

1. Web Designer: They all seemed to figure out I do web design stuff and sell these services to others. Looks like that message is getting understood quickly and correctly.

2. Intended Audience Confusion: I never noticed this before but the site is rather confusing in how it addresses its audience—Drupal Enthusiasts vs. People Needing Web Design services. One initial actionable would be to move the Drupal stuff out of the front page some and keep those Drupal references to services and blog page.

3. 2-Columns Unbalanced: A few of the reviewers commented about the two-column look and how it would be good to not have this look everything. In fact, I never planned to have a single layout and was intending to do this. Look like I need to do, especially on some pages like the front page and portfolio where the current layout doesn’t make much sense.

4. Improve Prominance of Contact Methods: It was interesting that one or two reviewers mentioned that it would be good to give a more direct contact instead of through the contact form.

The Elephant in the Room: Should I Put Public Examples of Price Quotes?

“If I knew how the prices compared to other web designers I may return.”

“The only thing I could not find was an estimate of the cost. I would have liked to get an idea of how much it costs without contacting someone. I can see this being shown as a sample. Say a web site with 1 page costs this, a web site with 4 pages costs this. Just an approximation to give a reader a general idea. “

“I am one who believes instead of just stating that someone can get a free quote that at least a general price range is provided. I think some people don’t even necessarily want to make an effort unless they see a range they like right away.”

“If I knew the cost range, I would be definitely coming back and referring people. Just the other day someone asked me to recommend someone who can rebuild an out-of-date web site. This sounds perfect, but without providing a price range I am not sure my recommendation would be complete.”

In a world where people can find the price for everything, it’s no surprise that several reviewers noted that they would have liked to have a general price range listed instead of a contact button for requesting a quote. I suppose from a consumer’s perspective, I would too. Unfortunately, I don’t know many other web design websites that provide this information publicly either. I suppose this is an idea to consider long-term for some of my more “repeatable” type sites. But overall this idea of publicly showing the price is likely something to consider in a different post.

A Word to the Wise: Some Conclusions on the Limits of this Kind of User Feedback Testing

Ok. Let’s be honest here. The users that reviewed my site were choosen from the people looking to do small tasking in the mechanical turk program. These are not really the ideal people to review my site. That said, they still provided fresh eyes and their comments are helpful in bringing light to certain missed issues.

On my client project, our initial task via this service was just a general impression and test of comprehension. The next stage is to see about real usability questions: Are some of the webforms logical? Can people go through certain internal tasks and complete the normal use case for X role? Etc.

In the end, this is just the first step in what could be a much deeper and wider range of user feedback and usability testing. But for now, at this price, it’s definitely something to bring new ideas and fresh impressions to the table, which really can’t hurt any project.

In today’s world of more and more technology, there is still a place for real humans looking and giving their opinions and perspectives on things. Sure, experts can give you guidelines and they have trained their eyes to isolate the pain points in any design or web applications. But in the end, we got to get outside of the office, outside of our tendences to see what we see: we got to ask normal people.

Feedback Army is just one service among many offering usability testing and feedback. Some others include “TinyMyUI:http://www.trymyui.com/, Easy Usability and User Testing. These provide videos of the usability testing with vocal report of the user experience. These are obviously better than quick feedback via text questionaires. But in the end, cheap and easy makes us more likely to use it. The Drupal code for Mechnical Turk shows a lot of potential in getting usability testing tools built for and with Drupal.

For me, I suppose my next step is jump directly into MTurk and see about posting some manual tasks so I can better understand the system, the community and its usefulness for me as sometimes-designer, increasingly-usability-conscious drupal-dev.

Let’s make your site and Drupal CMS easy to use. Do tests. Get feedback. Fix the pain points.

Tags: usabilitywebsitetestinguser experiencedrupalDrupal PlanetPlanet Drupalmechanical turkamazonfeedback

Mark Koester
@markwkoester
Mark has worked on Drupal since the early days of D6. He is passionate about open source as well as entrepreneurship. When he isn't building, he enjoys traveling and speaking one of his many foreign languages.
Chengdu, China

• var switchTo5x=true;stLight.options({publisher:'dr-8d62520e-c444-86e2-d911-ab27da97627b'});