Inside Sun’s Usability Lab…

usabilityI observed a usability study today in Sun’s usability lab. Martin’s group is conducting a usability study on a number of new search ideas that are being kicked around. I began program managing’s search a few months ago, so I figured this was a great opportunity to not only learn more about the science of search (a LOT to learn), but also learn more about usability studies and user input on search. I’ve initiated a number of usability studies on various projects, but never watched them real-time.

The study consists of several one hour sessions with a single interviewer asking a single interviewee scripted questions. The types of questions depend on the type of study (branding, functionality ease of use, new concepts, etc.). In this case we are interested in gathering ease of use information for existing functionality as well as input on some new concepts that are not yet deployed.

The person conducting the interview is a non-Sun person and clearly states this to the users up front. The idea is they’ll feel comfortable stating their honest opinions vs stating what they think we want to hear. She did a great job of drilling down on responses to ensure the users comments were clearly understood, but avoided influencing their opinions with repetitive or leading questions like they do in the movies until some poor sucker finally confesses to a murder s/he didn’t commit.

The lab is a lot like what you see in the movies during an interrogation scene – bright room with the interviewer/interviewee on one side separated by a sound-proof one-way mirror with observers sitting in a dark room on the other side. Once I got over the “Wow this is cool! What does this do?” phase of getting familiar with the lab and began focusing on the test in-progress, I found the user responses quite interesting, surprising and at times rather humorous. For the most part, the validation of ideas was good, but I found the negative responses far more intriguing.

One of the specific points this study validated was users often abandon site navigation if search results are present. The study also validated that a few features of’s search that we suspect are ambiguous are in fact so and via the study we were able to gather a few ideas on how to improve them.

Overall, observing a live usability study was an enlightening experience. It was good to observe a non-Sun person using our stuff and nice to see search meeting the expected needs for the most part.

  1. #1 by Muthu Online on October 7, 2005 - 1:35 am

    <font size=”3″ face=”Arial, Helvetica, sans-serif”>Interesting post on Usability
    Lab Testing! Here’s some more food for thought. </font>

    <font size=”3″ face=”Arial, Helvetica, sans-serif”>Test participants tend
    to feel comfortable giving negative feedback to a person who has not designed
    the application. As a thumb rule, Usability tests are not conducted by the
    designer who has designed the application. </font>
    <font size=”3″ face=”Arial, Helvetica, sans-serif”>Test participants must
    be informed upfront that <strong>ONLY the Application is being evaluate</strong>d
    and not their ability perform tasks.</font>
    <font size=”3″ face=”Arial, Helvetica, sans-serif”>Test participants are
    also asked to “think aloud” ( voice out their thoughts)</font>
    <font size=”3″ face=”Arial, Helvetica, sans-serif”>The Test facilitator
    should be neutral, probe a lot and must not ask leading questions or be intimidating.</font>
    <font size=”3″ face=”Arial, Helvetica, sans-serif”> Recruiting the right
    kinda Test participants is crucial as it will directly affect the test results.</font>

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: