Marianne Kernohan

Usability Testing: When to use remote usability testing

20 January 2014

Remote usability testing offers greater audience diversity and higher fidelity to real-world user motivations than in-person methods (but should not entirely replace them). We recommend using ‘Moderated’ testing for understanding an issue and “Unmoderated” to get large volumes of data.

Introduction – to remote usability testing

Traditional in-person usability testing involves an evaluator and a participant being in the same location at the same time and looking at the same screen. The participant in the usability testing session is then asked to perform certain tasks and vocalize their thoughts. Most usability professionals will tell you that usability testing is an invaluable tool in identifying and understanding usability issues. The method does, however, have certain weaknesses – such as:

  • Personal investment - Participants are often not personally invested in the tasks, no matter how hard you try and recruit relevant groups (i.e. they are not really considering buying this brand of car, they are just pretending).
  • Geographical limitations - The evaluators have to be in the same room as the participant, which means any truly international usability testing will be quite expensive.

Remote usability testing is a technique which seeks to address these issues. It does this by observing real-world users as they voluntarily use your site for their own purposes. Remote usability testing usually involves:

  • Participant and evaluator being in different locations
  • Observation and communication being mediated by technology

Benefits – of moderated remote usability testing

Although each method of remote usability testing has different characteristics and is best suited to answering different kinds of questions (as discussed in the next section), they do all share some benefits. Here are the benefits of almost all types of remote usability testing:

  • Audience diversity - You can easily run usability testing sessions with participants from different regions and/or countries.
  • Technical environment - People will be using their ‘real-life’ devices in their normal environment. This could, of course, have a large impact on participants’ behavior.
  • Personal investment - You can recruit participants as they arrive on your site, or are about to start a task you want to study. This means they will be personally invested in what they’re doing and have ‘real world’ questions that they want answered.

Note – Ethnio is a tool (currently free) which allows you – with the addition of 1 line of JavaScript to a page – to present potential participants with a DHTML pop-up screener layer (that cannot be blocked by a pop-up blocker). You can control the questions in the screener and users’ responses are immediately available for follow-up.

Types – of remote usability testing

There are 3 main types of remote usability testing. Each has their strengths and weaknesses and may be appropriate to different projects at different times. The 3 types of remote usability testing are:

  • Moderated - Similar to in-person usability testing, this involves the evaluator watching the participant complete tasks using a screen-sharing tool, and asking them to vocalise their thoughts (over the phone).
  • Unmoderated – user reported - Participants enter feedback and answer page-specific questions in a browser frame as they navigate the web site. The results are then collated (automatically or manually).
  • Unmoderated – automated statistical - Evaluators define the idealised, error-free performance of a task. Data is then automatically collected on users’ actions and compared to this idealised model (either manually or automatically).

Making a choice – selecting a remote usability testing method

A good general guide is that Moderated is usually the best choice of remote usability testing method if you want to develop a deep understanding of participant behaviours.

Due to Unmoderated’s potential to allow you to run 100s, if not 1000s of remote usability testing sessions quite cheaply, it is often used to identify potential problem areas (for further investigation through Moderated methods), or illustrating the scale of an issue identified during Moderated sessions.

Please note: ‘Unmoderated – automated statistical’ remote usability testing is best for simple and easily-definable tasks.

Moderated – issues with remote usability testing

As we have discussed, Moderated remote usability testing is a great method for developing a deep understanding of participants’ behaviours. There are, however, some particular issues concerning this method of remote usability testing which you will have to consider before planning any remote usability testing session:

  • Tools for screensharing, recording & communicating - It is important to find tools that give you the data you want, but it’s equally as important to find tools which are easy for the participants to install and use. Another factor to consider is the bandwidth which a tool requires, as this may limit your potential participants.
    Note: Most remote usability testing practitioners relate good experiences of using ‘Reverse Morae with GoToMeeting’, which allows you to use Morae’s features to tag participant behavior).
  • Facial expressions & body language - Non-verbal cues are crucial in real-world usability testing, as they often portray a participant’s un-vocalised thoughts/attitudes. Unfortunately, these are normally not present in remote usability testing.
  • Personal connection - It is often harder to establish rapport and trust with a participant during a remote usability testing session. For this reason, it is often worthwhile to spend a little time at the beginning of the session introducing yourself and ‘breaking the ice’.
  • Supporting users - Helping a user to overcome a problem (for example: when they don’t know what to do next) can be difficult within a remote usability testing session. We have found detailed instructions are better than taking keyboard-control (because this breaks the unity of the participant’s experience).

Unmoderated – issues with remote usability testing

When using Unmoderated remote usability testing methods, it is important to try and capture how successful the site was in supporting participants’ goals. The issue under consideration is this: participants might be able to find the page which should answer their question, but does the page itself actually succeed in this goal?

One of the best ways of dealing with this is to ask participants a question, such as: “Which car has the largest trunk space?” We would advise that you never rely solely on Satisfaction Surveys and always include such questions in your Unmoderated studies.

Please note: ‘Unmoderated – automated statistical’ remote usability testing is best for simple and easily-definable tasks.

Incentives – issues with remote usability testing

We have found that the most convenient incentive to offer participants is an Amazon gift certificate. These certificates have the advantage of covering a wide range of goods and only requiring the participant to provide their email address.

Summary – remote usability testing

Remote usability testing offers several advantages over in-person usability testing. These include a greater degree of audience diversity and a higher fidelity to real-world users’ technical environments and motivations. Moderated remote usability testing is best-suited to developing a deep understanding of an issue, whereas Unmoderated is best for getting large volumes of data.

The Moderated method of remote usability testing can not, however, be considered to completely replace in-person usability testing because of its limitations in the personal interactions between evaluator and participant. We do, however, believe that remote usability testing represents a very cost-effective opportunity to explore a site’s usability.

Start your project

Got a project in mind? Get in touch.

+44 (0)20 7168 7526

Back to top