The Role of Usability Labs in System Design
This item is not the definitive copy. Please use the following citation when referencing this material: Dillon, A. (1988) The role of usability labs in systems design. In: E. Megaw (ed.) Contemporary Ergonomics 88. London: Taylor and Francis, 69-73.
The issue of usability is a central concern for contemporary system designers and a major focus of research in the domain of HCI. In an attempt to evaluate the usability of products some companies have invested heavily in the the development of so-called "usability labs". Consisting of sophisticated video recording equipment and observation facilities, these laboratories may well be expected to provide insight into the process of interaction that would otherwise remain hidden. Is this in fact the case? Are usability labs the universal panacea for the problems of evaluation? The present paper outlines the advantages and details the limitations of such facilities and argues that the problems lie less with the laboratory and more with the evaluator.
Incorporating human factors within the design of modern computer systems is now seen as a desirable goal by most major IT companies. In recent years, this drive towards more usable or user-friendly product design has increased the awareness in the human factors community of the need to find ways of fitting relevant knowledge into the often disorderly and compromised design process of modern products (Hannigan and Herring 1986, Galer and Russell 1987). Ideally, design proposals and evaluations should occur iteratively. However in reality the discipline of HCI has not yet reached the stage where suitable techniques exist for early evaluation and testing is usually left until late in the design process when a working system exists for human factors specialists to assess (see Bertaggia et al 1987).
Given the recently perceived importance of designing for usability and the preference of the discipline for evaluation rather than prescription a number of companies have invested in the development of so-called "usability labs". Typically these are employed for formal evaluations of new or revised products and on the basis of recent trends would appear to be growing in popularity.
2. WHAT IS A USABILITY LAB?
Obviously no two labs are identical. The usual laboratory consists two adjacent rooms connected by door and a one way mirror allowing experimenter observation of users. Subjects or "participants" in the investigations sit at a terminal in a room usually designed to resemble a normal office environment. Video cameras are positioned so as to capture images of the screen, the keyboard, the user's face or gross body movements and possible documentation usage. Users' verbal elicitations can also be captured.
The investigator usually sits in the other room which contains a variety of recording and image mixing equipment. Facilities exist for controlling the cameras from this room (e.g. angling, zooming and focussing) therefore ensuring the ability to select and capture interesting events as they occur.
The output from these labs usually takes the form of a mixed video recording that displays timestamped keyboard and screen activity simultaneously and incorporates further events of interest that may have occured beyond these two views. Further descriptions of labs can be found in Bertaggia et al (1987) and Youmans (1987).
3. THE VALUE OF USABILITY LABS
The advantages of video playback of user interactions combined with apparently unobtrusive data collection, theoretically add up to an investigative methodology of such power that one could imagine the emergence of rigorously "scientific " usability metrics and resultant design recommendations that will forever solve the problem of what constitutes usability.
By recording images of the screen, keyboard, manual or documentation usage and possible verbal and facial reactions these labs afford the capture of significantly more data than any trained observer or group of observers could hope to record. Coupled with this, they allow the experimenter to view the session repeatedly and to rate any number of interesting phenomena without fearing data loss or reduction.
The lab also offers a superb means of presenting feedback to interested parties. The provision of suitable feedback to interface designers is undeniably important, though not well understood at the moment. Visual evidence of user difficulties and complaints makes it possible to simply demonstrate the importance of human factors in user performance with, and acceptance of, a system.
Thus labs offer three distinct advantages to my mind over alternative techniques: They maximise data capture, allow unlimited reviewing, and in themselves afford a direct means of feedback to designers.
Even though labs offer such large scale data capture without any need for a priori selection they do not come complete with any unique methodology for assessment . Ultimately they are only data capturers and any analysis and interpretation of this data will eventually require skilled human factors personnel.
Furthermore they are expensive. Costs vary and lab developers are never quick to discuss finance but one must assume large capital input is necessary to provide the standard two-room set up and necessary video recording and mixing equipment.
Once a lab is developed it becomes virtually rooted. Products and participants must be brought to the testing area for usability evaluations. While this may not seem too distinct a disadvantage,a transportable means of system evaluation ( such as a trained ergonomist!) would be more desirable.
It is not yet clear how applicable the lab based approach is for evaluating products at earlier stages of design. Intuition suggests that where products are not approaching completion the usability lab will be of little use for assessment.
4. BROADER ISSUES OF USABILITY EVALUATION
4.1 The nature of Usability
Beyond the purely obvious pros and cons of labs are issues pertaining to their use which are of considerable importance to the human factors community. As mentioned earlier, the output of labs consists typically of a video recording of user interactions with a system.. However this in itself is of little use without a clear understanding of how to proceed from here. To make sense of this output i.e. to form a reasoned conclusion of the usability of the product under assessment, one must know what are the important criteria upon which to base an assessment.
The literature on HCI is full of attempts at system evaluation but as yet no formal standards have emerged or metrics been arrived at which can be usefully employed to gauge real world usability. In their absence evaluators are left to rely on their own wits, knowledge and experience to interpret from the video recording where the system fails, where it is good (an often overlooked point) and how it can be improved. Thus the usability lab is less a means for assessing the user-friendliness of a system and more an augmentor of the experimenter's data capturing powers.
Furthermore the very term "lab" is misleading. It suggests formalism, empiricism and the quest for knowledge. Few experiments are really carried out in current usability labs. Investigations into what constitutes usability are not their raison d'etre. They are more often than not places of quality assurance where science is in short supply. This may seem unnecessarily harsh but it is important to understand that interface evaluation is not a precise science. In many ways it remains an art. The artists, in this case, experienced human factors specialists, will identify aspects of the user-tool interaction that are symptomatic of usability failings. But it is hard to predict in advance what these will be or instruct others in what to look for. Proponents of labs may espouse the virtues of their own "methodologies" but the simple fact of the matter is that agreed standards and procedures do not yet exist. Criteria of acceptability vary. In short usability is what you define it to be!
4.2. Lab-based observations
Criticisms of laboratory studies of human performance and behaviour have a long history in mainstream psychology (see e.g. Christensen 1977). The very design of many of these labs suggests a desire to create a natural or real-world atmosphere to the interaction under investigation. However it is not clear how well findings from such studies relate to user behaviour in his/her place of work. Similarly much of the interest in such usability evaluations is centred on a narrow range of issues to do with the interface e.g. do users access the help facilities frequently or navigate correctly through several layers of menus. These are not necessarily the issues that will have the greatest bearing on product usability in the real world. However, neither labs or human factors practitioners are yet sophisticated enough to cope with such issues.
Contemporary I.T. product design is a complicated process. The emergence of usability labs can be best appreciated as an acknowledgement by many companies of the importance they now place on incorporating human factors in their systems. While this is progress of a sort, there is little evidence to suggest that labs are affecting better designs. Furthermore formal operationalisation of the usability concept has proved difficult and the onus remains firmly on the individual evaluator to define criteria and methods for assessment. The lab is merely an observation room.
Labs do have their uses. They also have their drawbacks. Investigation is needed to identify how best to exploit their potential and what alternatives are needed to tackle issues they cannot address. Without such knowledge the human factors community may justifiably be accused of hiding behind its own techniques
Bertaggia, N., Novara, F., Dillon, A. and Bonner, J. (1987) The evaluation of products using the methodology with proposals for product improvement. Working Paper A.5.3a, ESPRIT Project 385 - HUFIT
Christensen, L.B., (1977), Experimental Methodology (Boston: Allyn and Bacon).
Galer,M. and Russell, A.,(1987), The presentation of human factors to designers of IT products. In Human-Computer Interaction: INTERACT '87, eds. H.J. Bullinger and B.Shackel, Amsterdam, North Holland, pp.11-16.
Hannigan, S. and Herring, V., (1986), The role of human factors in the design of IT products. Deliverable A12b. ESPRIT Project 385 - HUFIT.
Youmans, D., (1987), Using video in the design process, IEE Colloquium on Evaluation Techniques for Interactive System Design: Digest No.: 1987/83.