Usability Testing: How To Statistically Compare The Usability Of Different Designs

Introduction – a useful application of usability testing

Usability testing is mostly used to identify and explore usability issues on a site. This normally employs the ‘talkaloud’ method of usability testing, where a user is asked to complete a task and then verbalise their thoughts during the usability testing. Such a use of usability testing is very valuable and almost all significant web development projects will conduct this kind of usability testing before launch.

Usability testing can, however, be used in a different way. The ‘benchmarking’ method of UX research allows you to get a statistical measure of a site’s usability during a usability testing session. This is a very powerful idea, because it allows us to use usability testing sessions to statistically compare the usability of different designs and/or sites.

‍

How it works – using usability testing to get a measure of usability

The generally-accepted definition of usability is that provided by ISO 9241, which states that usability is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”. So, in order to use usability testing to get a quantification of a site’s usability, we need to find a way of statistically measuring effectiveness, efficiency and satisfaction.

In the ‘benchmarking’ usability testing we have done for some of our clients, we have measured these on a per-task basis. Here are the ways we have found to statistically measure these parameters during a usability testing session:

• Effectiveness – This is usually the simplest one to measure during usability testing. For most tasks, you will be able to use a pass/fail measure of whether the user has managed to complete the task successfully. The percentage of users that manage to complete a task successfully therefore becomes a measure of the design’s effectiveness.
• Efficiency – A useful way to measure this characteristic can be timing the user on a task during the usability testing. The average task completion time of all the users participating in your usability testing therefore becomes a measure of efficiency.
• Satisfaction – The easiest way of measuring users’ satisfaction is normally in the form of a questionnaire. This questionnaire is typically administered after the usability testing session has ended and asks the user to rank their experience of the site (along a number of parameters) from 1 – 5 (1=poor, 5= good). The questions can relate to the entire site experience, or ask specific questions about each task.

Using these simple techniques you can get data which tells you (in a quantified way) how usable the site was found to be during usability testing. It should also be noted that during ‘benchmarking’ usability testing sessions, users are not encouraged to verbalise their thoughts, as this interrupts their thought-process and slows down task completion.