🧰 Design Methods Toolbox

# Moderated Testing

# Phase: 🛠️ Problem solving
Focus: Test

IN BRIEF

Time commitment: Varies widely according to test method, but always more time than unmoderated testing
Difficulty: Moderate/difficult
Materials needed: Goals for outcome, moderator guide, users to interview, location (physical or virtual), interviewer, notetaker, notetaking tools
Who should participate: User experience designers, product/project owners, community specialists
Best for: Gaining in-depth insights about a solution's effectiveness that cannot be gathered through unmoderated or passive/quantitative methods

"Have the mindset of testing to learn, not testing to validate." Brendan Boyle, IDEO

# About this tool

Moderated testing isn't a tool in itself, but rather any form of testing that takes advantage of active moderation by an interviewer. A number of these are already featured in this toolbox (and please open an issue or make a PR if your favorite has been excluded so far; this toolbox is a living document!), but examples include the following:

  • General usability testing (not to be confused with user testing; see below)
  • Card sorting (this could be moderated or unmoderated)
  • Think-aloud testing
  • "Guided tour" testing
  • Empathy interviews

Note that when it comes to moderated testing, the area of overlap between tools categorized as empathy or information-gathering tools, and tools considered testing tools, is considerable and the distinctions largely arbitrary. Many of the information-gathering tactics categorized in this toolbox as empathy-focused problem seeking tools work just as well when used in a moderated setting in conjunction with a wireframe/prototype and task prompts.

It may be helpful to think of moderated testing as the most rigorous option in the spectrum that begins with data gathering using passive analytics and moves through unmoderated testing.

Moderated testing has a variety of pros:

  • It shines in qualitative lines of questioning, because you can ask personalized followup questions and change the direction of an interview if/when needed
  • It enables you to act on nonverbal or contextual clues and gives you a valuable opportunity to read between the lines
  • It staves off the danger of wasting entire tests due to poorly worded, confusing or non-applicable questions; you'll know reasonably quickly whether you're going down a useless path

However, moderated testing also has some cons:

  • It's expensive in a variety of ways: Time spent recruiting, time spent interviewing, monetary cost of rewarding interviewees, time spent collating and analyzing responses
  • It doesn't scale well
  • It can be easy to introduce your own biases without realizing it through your own nonverbal cues or riffs on a question script
  • If undertaken remotely, it requires coaching the interviewee on installing teleconferencing tools, consenting to recording a browsing session or audio conversation, etc

For these reasons, moderated testing is generally more appropriate for qualitative testing than quantitative testing, which can often be succesfully undertaken without the need for moderator intervention or followup. Where moderated testing particularly shines is in usability testing (not to be confused with the more generic term of user testing). Often an observational exercise, usability testing evaluates how useful and usable a solution by watching and discussing how a user interacts with it. By asking your user to complete a task, observing them as they do so, and following up by asking for details on their thoughts and feelings experienced during the task, you're able to gather rich insights both on what works well and what needs to be addressed in future iterations of your work.

In any case, whether you're conducting a moderated or unmoderated test, you'll want to keep a few things in mind:

  • Make sure to expressly indicate the metrics you'll be measuring before even writing a test plan or test questions. Is the key item you're seeking information on successful completion rate, time spent on a specific task, or a quantitative or qualitative measure of satisfaction?
  • Make sure you're considering both "extremes and mainstreams" (ordinary users and edge cases), both when you're writing up your test plan/questions and when you're recruiting for participants.
  • Make sure you're hitting the sweet spot in terms of number of participants. For qualitative testing, a minimum of five participants helps reduce the effects of outliers or statistical noise; more participants can add more value, but keep in mind that there will be a point of diminishing returns simply due to the amount of effort required to capture the information gathered during the test.

← Principle definition →