User testing
For a product or service, the acid test of accessibility is asking real users with disabilities to attempt real tasks using it.
Benefits
- Reveals whether your product or service works as you intended;
- Works for websites, software applications, consumer products, and much more;
- Helps designers get into the mind of their users;
- Reveals the effect of design features on accessibility, usability and task performance;
- Produces a prioritised list of issues and recommendations for improvement;
- Short video clips are great for demonstrating issues to senior management.
Even if you have followed accessibility guidelines, it can be very difficult to be sure that a product is fully usable by people with disabilities in practice. The best way to find out is to do some real user testing.
In a CFIT user test, you can observe users with disabilities carrying out typical tasks using a product such as a website, a software application or a piece of equipment. Any difficulties, confusions or frustrations will become immediately apparent. A CFIT accessibility consultant will help you understand exactly how the design or implementation contributes to these problems and what you can do to fix them.
How does it work?
Here is an overview of the CFIT user testing facility as used for a typical website user test.
The user environment
The user test participant (1) sits in a typical office environment within the testing room which is controlled for sound. The test facilitator (2) sits with the user, explaining the tasks, taking notes and critically observing the user's interactions. The test is conducted using a standard PC (3) with assistive hardware and software. Dedicated user test recording software, together with discreet cameras and microphones capture and record every aspect of the user testing session for later analysis
The Observation environment
Observers can watch the test in real time from the comfort of our observation room couches. The video from the user's monitor (6) is displayed on a flat screen TV (4) while a second signal from the camera and microphone (7) shows the user's gestures, facial expressions, body language and vocalisations on a television monitor (5). Through these links, observers can see everything that the user does and says, as well as the interaction between the user and the facilitator.
What do you get?
A typical user test consists of 8 separate user sessions of 1 to 1½ hours each. Each user carries out a set of realistic tasks which have been agreed beforehand with the client. These will usually include the most common tasks for which the product is used, as well as the most critical tasks and any tasks that CFIT's experts anticipate may cause problems for users. Tests are carefully designed and run so as to yield the most realistic user behaviour and therefore the most valid results.
CFIT recruits representative users to suit the test. Recruiting 8 different users enables us to cover a broad range of disabilities and assistive technologies. It also allows us to include younger and older users and people with different levels of experience. This results in a more representative sample of attitudes and approaches.
Most tests take place at CFIT's purpose-built user testing facility in Drumcondra, Dublin, but we can test on-site if necessary. If individual users are unable to travel or if they use assistive technologies which cannot be transported or reproduced, we can run the test in their home, workplace or rehabilitation centre. Extended field trials can also be carried out in this way.
We strongly encourage clients to come and observe the tests. Observing user tests is one of the best ways to gain a first hand understanding of what accessibility really means, beyond the technicalities of any guidelines. Designers and developers in particular can get huge benefits from the insights they gain from observing users. So much so, that we have created a separate observation area for that purpose in our Drumcondra facility. Using a wide screen TV and a small video monitor, clients can watch and listen to the user tests via a remote link without disturbing the users in their tasks.
After we have analysed the results of the user tests, the findings will be presented to your project team by a senior CFIT consultant at a half day workshop. Digital video recordings of user sessions are used to illustrate key issues. The emphasis is on building an understanding of how the design of the site contributes to users' difficulties and what practical steps you can take to alleviate these problems.
What happens next?
The user test will give you an understanding of what it is like for people with disabilities to use your product or service and where problems arise. You may decide to combine the user tests with a full accessibility audit, to give you a complete picture of what needs to be done to achieve full technical accessibility and greater ease of use for all users.
CFIT can continue to assist you as you work on implementing the suggested improvements, by providing flexible consultancy, support, mentoring or 'help desk' facilities according to your needs. This will help you to quickly grow your own in-house accessibility expertise.
What does it cost?
An 8 person user test at our Drumcondra facility costs €4,600. A 4 person test costs €3,300. This covers planning and preparation, recruitment, running the test, payments to the users, video recording, analysis of results, report compilation and presentation at a half day meeting. Tests can be arranged with more or fewer users, depending on your needs.
As NCBI is a registered charity, we do not charge VAT, so this is the final price.
Case studies involving user testing
How CFIT’s auditing and user testing helped welfare.ie win the eGovernment Accessibility award
User trials of RTÉ digital broadcasts