International usability testing can benefit from: high-level task definition; locally-informed participant recruitment and locally-delivered moderation & analysis (supported by a thorough Testing Plan).
Why international usability testing?
Usability testing is a very powerful tool because its findings can be applied to the wider target audience (when the usability testing sessions are well planned, moderated and analyses). Critically, however, experience tells us that usability testing findings do not usually apply across very disparate user groups. This is why, for instance, an online retailer might run separate usability testing sessions for loyal existing customers and potential new customers.
It’s hard to define exactly how different a target audience needs to be before it justifies its own set of usability testing sessions. But most experts would agree that if 2 audience groups are likely to have significantly different users, goals and/or contexts of use, then separate usability sessions may be useful. Thinking along these lines, most people would probably agree that an audience located in a different country (with its own language and culture) is likely to represent a distinct group worthy of its own usability testing session.
2 main types – of international usability testing
When considering an international usability testing project, there is normally a choice between lab-based and remote methods. Each has its own advantages, but the principal differences can be summarised as follows:
- Lab-based (international usability testing) – More expensive to run and harder to organise. However, these international usability testing sessions do generally offer the best feedback on participants’ behaviours and attitudes.
- Remote (international usability testing) – Sessions are easier to arrange and cheaper to run, which allows a greater number of participants within an international usability testing study. However, some important participant feedback may be lost (through lack of facial and body cues, for example).1
In this article, we will focus on some of the issues with lab-based international usability testing. Our discussion of these issues has been drawn from our experiences of conducting numerous international usability testing studies. Although we would not suggest that the list of issues is exhaustive, we nonetheless hope that they will present a useful guide for anybody embarking on an international usability testing project.
Task selection – for international usability testing
It is wise to be very careful when defining the tasks for an international usability testing project. Time and again, we have found that customers in different countries often have different priorities when completing a task (such as researching a product purchase, for example).
These priorities could, for example, be due to product positioning or local market conditions. An example is provided by the automotive industry, where ‘fuel economy’ is far more important to UK-consumers than their American counterparts (because of the UK’s higher cost of fuel).
In general, we would recommend that international usability testing sessions follow these guidelines:
- High-level tasks – Use high-level tasks within an international usability session, which participants can then interpret for themselves. For example, asking a test participant to “find the right product for you” allows them to use their own priorities.
- Low-level tasks – Different types of information and/or functionality will be important in different countries. Therefore, we would usually only use very specific tasks (such as: “What is this vehicle’s fuel economy?”) when we know the task is a high priority for a country’s consumers, or representative of a wider task.
Task selection is, of course, critical in all forms of usability testing, but we have found it particularly important with international usability testing. Before the international usability testing session, it is always a good idea to talk to people with local knowledge about task definitions (whether in-company staff or external usability consultants). This will help make sure that tasks are relevant and nothing important has been missed.
Participant recruitment – for international usability tests
It’s vital to a make sure that a usability testing session’s participants truly reflect one’s target audience. Ensuring this is even more important – and challenging – with international usability testing. We would recommend that you always try and use local knowledge to define target audiences. This is because there may be more subtle factors at work within a country’s market than a foreigner may be able to appreciate.
For example: we recently helped run international usability testing sessions in a country which had a significant immigrant population. We decided to run usability testing sessions dedicated specifically to this audience, because local research suggested that their goals, language skills and means of access were all likely to be significantly different to other target groups. The results of the international usability testing proved this to have been a good decision and provided very valuable information.
It is also, of course, the case that local knowledge is very valuable in making sure you ask the right kind of questions to recruit the right participants.
Within the UK, for example, we may know what Newspapers our target audience tend to read – but we would struggle to answer these questions within a German context.
Moderation & analysis – international usability testing
How international usability testing sessions will be moderated and analysed is one of the most important issues in any project. This is because it will have a large impact on the international usability testing project’s success. The main options are usually:
- Central expert – All sessions within an international usability testing project are moderated and analysed by a single usability expert.
- Local expert (or company staff) – Sessions are moderated and analysed by a local usability expert (or properly trained local employee).
The ‘central expert’ approach has the advantage of ensuring consistent method and analysis across all the international usability testing sessions. Unfortunately, the ‘single expert’ approach to international usability testing tends to have a higher risk of missing subtle (yet important) linguistic and/or cultural cues. This applies regardless of whether the international usability testing sessions are run in the expert’s own language (which also limits the pool of potential participants) or through the use of simultaneous translation.
In general, we prefer to use a use a local expert (or a properly-trained member of company staff) to run and analyse the tests. This overcomes a lot of the difficulties associated with the ‘central expert’ method (see above). It does, however, place a very great emphasis on having high quality local personnel and a precise testing plan for the international usability testing sessions. That’s why we recommend spending a lot of time on these 2 key elements in order to make sure the international usability testing sessions will address the project’s goals in an effective and consistent way.
Testing Plan – vital for international usability testing
A Testing Plan is crucial to any usability testing project, but it is particularly important in any project with more than one moderator/analyst. This is because the Testing Plan should serve as a complete blueprint on how to plan and run the sessions. This should feature all relevant information, including:
- Goals – Background on (and articulation of) the project’s goals.
- Task definition & participant recruitment – Rationale underlying (as well as full details on) these aspects of the usability testing project.
- Moderation & analysis – Full description of the techniques to be used, along with any required supporting materials (such as session videos, see below).
- Documentation – Complete details on how the usability testing sessions (as well as their results, analysis and recommendations) should be documented. Supporting materials may include an example report (see below).
Because international usability testing projects often involve collaboration between teams in different countries, the Testing Plan is an invaluable resource for helping toplan and run sessions in a consistently effective way within an international usability testing project. There are however, additions to a standard Testing Plan which can be particularly beneficial to an international usability testing project. These include:
- Video (from central expert) – It’s usually much better to demonstrate what you want from your project partners rather than to simply describe it. That’s why we would recommend providing an edited video of either your pilot testing session or the actual sessions themselves (if they have already been conducted). Within these videos, attention should be drawn to particular issues which you would like local partners to focus on.
- Documentation (from central expert) – The best way to get the documentation you want is to provide local partners with a real-world example report which has the desired structure, format and level of detail.
Both of the above can be used to supplement a Testing Plan and help it to better support local partners in delivering an international usability testing project’s goals.
An additional means of making sure that international usability testing sessions will be run in a consistently effective way is to ask local partners to provide a video of their pilot testing session. This will allow the central team to review the way in which their local partners are planning to run the test and allow for any misunderstandings of the Testing Plan to be addressed.
Summary – remote usability testing
Within lab-based international usability testing projects, it is vital to ensure that local participants interpret high-level tasks according to their own priorities and that very specific tasks are only used in a tactical way. One should also remember that local knowledge is vital in participant recruitment, as there may be important factors impacting this process’ efficacy which foreigners will not be aware of.
The moderation and analysis of international usability sessions is often best carried out by local experts (or well-trained company staff). This will help ensure that nothing is missed due to language or cultural factors. In order to support this model of international usability testing, a comprehensive Testing Plan should be provided – including a video demonstrating an example session and a sample piece of documentation. Finally, the local moderator should provide a video of their pilot usability testing session in order to show their understanding of the Testing Plan.
1 Note: For a discussion on the differing methods that can be employed in remote international usability testing, please see the following article: Usability Testing: When to use remote usability testing.