Sheryl Burgstahler
University of Washington
Disabilities, Opportunities Internetworking and Technology
Tracy Jirikowic
Disabilities, Opportunities Internetworking and Technology
Beth Kolko
Department of Technical Communication
Matt Eliot
Department of Technical Communication
Address correspondence to:
Sheryl Burgstahler
(sherylb@u.washington.edu)
DO-IT Program
University of Washington
Seattle, WA 98105
206-685-3648
ABSTRACT
This paper reports on a collaborative project between a university and technology corporation that explored computer software usability and accessibility with older adults and individuals who have disabilities, some of whom use assistive technology. The project history and overall goals are described. In addition, the methods and results from usability/accessibility studies involving individuals with visual impairments, people with mobility impairments, and older adults are presented. The implications of these findings are discussed relative to the notion that in order for information technology products to be considered "usable" (i.e., efficient and easy to learn and use), they should be universally designed to meet a wide range of user needs, with or without the use of assistive technology. Suggestions for future research, as well as for promoting the consideration of the needs of consumers with disabilities during product design, development, and usability testing are also discussed.
Electronic and information technology products hold significant promise for all individuals to be better informed, able to communicate with greater numbers of people, and more successful in school and employment. This potential may be even greater for individuals with disabilities, many of whom report that information technology, including the Internet, has had a significant impact on their lives (DO-IT, 2004; Hasselbring & Glaser, 2000; Success Stories, 2002; Taylor, 2000). Advancements in computer technology have opened doors to educational and employment opportunities that were once closed to individuals with disabilities. Most individuals with disabilities who are employed use computers at work and as a group rely on a wide variety of assistive technologies (National Organization on Disability, 2004). However, individuals with disabilities are less than half as likely as their non-disabled counterparts to own a computer, and they are about one-quarter as likely to use the Internet (Kaye, 2000; US Department of Commerce, 2000).
Computer technologies have the potential to provide significant benefits to people with disabilities. Computer users with visual impairments can gain immediate access to content, no longer needing to wait weeks or months for printed information to be translated into Braille or audio recording. They can instantly access newspaper articles, reference materials, program information, product prices and purchasing options, and communication with associates without depending on the assistance of others. Those confined to their homes can shop for necessities, communicate with others, pursue academic degrees, and telecommute to their employment. Speech recognition technologies and other alternative input devices allow those with mobility impairments to bypass the standard keyboard and mouse or pen, to communicate informally or professionally with ease (Brewer, 2001). Yet economic and other factors contribute to putting people with disabilities disproportionately on the wrong side of the digital divide in this country. Additionally, those people with disabilities who do have access to computers and assistive technology still face challenges gaining full access to computing resources because of their inaccessible design (Burgstahler, 2003; Gunderson & Mendelson, 1997; Rowan, Gregor, Sloan, & Booth, 2000; US Department of Commerce, 2000).
Besides the economic issues that help perpetuate a digital divide, design barriers further impede equal access. Although assistive technologies (e.g., text-to-speech software, speech input, alternative keyboards) provide the means for most people with disabilities to access computers, Web pages, applications software, and tutorial programs are often designed in such a way that they erect barriers to individuals with disabilities both for those using assistive technologies and those not using them. For example, individuals who are blind cannot access Web pages and applications software with graphic images, unless the content within these images is also provided in a text-based form for text-to-speech software to read aloud. In addition, some products that include accessibility features do so in such a way that they are cumbersome for individuals with disabilities to locate and use. Additionally, Web sites are not always efficient and easy to use for individuals with disabilities. One study found that Web site usability was three times better for individuals without disabilities when compared to individuals with visual or mobility impairments (Coyne & Nielson, 2001). Many studies have found that most Web sites include inaccessible features (Schmetzke, 2001; Thompson, Burgstahler & Comden, 2003).
Two legal statutes have raised public awareness of accessibility issues in the United States. The Americans with Disabilities Act of 1990 (ADA) prohibits discrimination against individuals with disabilities in public programs and services. Although the ADA does not specifically mention information technology, the United States Department of Justice clarified that the ADA applies to Internet-based resources (Patrick, 1996).
Section 508 of the Rehabilitation Act of 1973 mandates that federal agencies develop, procure, and use electronic and information technology that is accessible to individuals with disabilities. As required by the law, the Architectural and Transportation Barriers Compliance Board (Access Board) developed access standards to which federal agencies must comply (Office of the Federal Register, 2000). The standards cover the purchase, development, and use of software, Web pages, videotapes and multi-media, telecommunications products, standard office equipment, and other electronic and information technology.
Although it is to the advantage of technology companies who wish to sell products to the federal government to comply with Section 508 standards, many have not yet addressed access issues in a routine and comprehensive manner. Such companies fail to sufficiently incorporate the needs of users with disabilities into the design process. Even for companies that make some efforts, consideration of design elements for people with disabilities often occurs separately from the overall design process and rarely involves actual users with disabilities. To assure that accessible products are created, users with disabilities and their accessibility issues should be included in all phases of product design and development. This paper explores how accessibility can be incorporated into usability testing processes as part of an ongoing product-development cycle.
The process of creating products that are accessible to people with a wide range of abilities, disabilities, and other characteristics is called "universal design." Universal design is defined by the Center for Universal Design at North Carolina State University as "the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design." (Center for Universal Design, 2002) At this Center, a team of architects, product designers, engineers, and environmental design researchers established a set of principles of universal design to provide guidance in the design of environments, communications, and products.
Typically, products are designed for the average
user, rather than with consideration for the broad range of
characteristics of potential users. The application of universal design
principles during product design has the potential to make products
accessible to and usable by more consumers. Computer hardware and
software developed with these principles in mind may eliminate or
minimize the need for assistive technology, and/or reduce compatibility
problems that sometimes arise between basic systems and assistive
hardware or software.
Anticipating specific features that will be accessible to users with
visual, mobility, hearing, or other types of impairments is difficult
without direct input or feedback from individuals with disabilities
themselves. To create fully accessible products, accessibility issues
must be considered at every phase of the development process. One
important time at which this should occur is during standard usability
testing, which is employed to ensure that new technologies are easy to
learn and easy to use.
Usability testing is employed by many industries to ensure that product design is usable by members of the targeted user population. The term usability has two distinct meanings within the current product-development milieu. The first definition of usability refers to the users' experience, to their ability to effectively perform tasks using a given product and to their satisfaction with the function of a product. The second definition of usability, actually a truncation of the term usability engineering, refers to an iterative testing and feedback process employed by designers and engineers wherein actual users are observed as they interact with specific feature(s) of a product. This usability process is often utilized during successive phases of product development, with the intention of making product use more efficient and attractive to customers.
The term usability was first popularized in the early 1980's, as focus shifted from programmer-computer interaction to user-computer interaction (Gould & Lewis, 1985). The usability design and testing process as it is currently understood contains four guiding principles intended to ensure that the needs and preferences of actual users inform the product development process. These principles call for the inclusion of actual users at every step of the design process:
Jakob Neilsen, arguably the most visible proponent of usability processes, articulated the following elements that comprise a definition of usability (Nielsen, 1996).
Usability testing is an effective use of time and materials (Nielsen & Landauer, 1993; Virzi, 1992). Research on usability is included in this article, in part, as an attempt to incorporate usability methods into accessibility research. At the same time, we would like to suggest that accessibility questions be incorporated into general usability methodology. Generally speaking, the insights gained from an iterative set of procedures help identify product features that are key to target user's experiences and to their sense of satisfaction. In the case of this study, usability testing has shown that designing for accessibility must go beyond hardware concerns, such as specific assistive technologies, and even beyond software accessibility concerns and incorporate broader design concerns such as information architecture strategies.
Currently, accessibility tests on new products are rarely employed and, when they are, they usually take place after standard usability testing is complete. If accessibility testing could be conducted throughout the design process, there would be more and better products available for people with disabilities. At best, current accessibility-testing practices often result in products with cumbersome accessibility features that are incorporated only after the primary design decisions about the products have been made. This approach typically results in a product that meets minimum accessibility standards, which "doesn't mean that it's easy to use, simple to learn, or supports efficient job performance (Coyne & Neilson, 2001).
Usability tests generally include individuals with characteristics (e.g., age) that are typical of the potential customers for the product. Historically, people with disabilities have not been included in the standard usability tests of mainstream products because, at least in part, designers do not routinely consider people with disabilities to be part of the ultimate user base. With the growing size of the disabled population in general, and elderly citizens specifically, producers can anticipate a growing market for accessible products. In addition to existing professionals in the usability field, colleges and universities that teach students how to perform usability tests do not usually teach students about how to include people with disabilities in such tests. Further research is needed to explore how people with disabilities might be included in usability tests. In the research discussed in this paper, the authors employed usability testing to answer the basic research questions that the Microsoft Accessible Technology Group had about the built-in accessibility features of the Microsoft (r) Windows(r) XP operating system. The procedures and results should benefit those conducting usability tests on other technology products.
To address issues related to software product usability testing by individuals who have disabilities and older adults, collaboration occurred between the Microsoft Accessible Technology Group, the University of Washington departments of Technical Communication and Computer Science and Engineering, and DO-IT (Disabilities, Opportunities, Internetworking and Technology). The Microsoft Accessible Technology Group, in support of Microsoft's corporate mission, enables people and businesses throughout the world to realize their full potential. DO-IT is a collection of programs that promote the success of individuals with disabilities in postsecondary education and employment, using technology as an empowering tool. In several programs DO-IT works with high school and college students who have a wide range of disabilities to promote academic and career success. The objectives of the collaboration reported in this article were to:
A central research activity of this collaboration was conducting usability and accessibility tests of Windows XP. The following section describes the subjects, methods, and results from this exploratory study.
Purpose and Research Questions
Microsoft sponsored this study to improve the usability and discoverability of the accessibility features built into the Windows XP operating system. Microsoft hoped that the usability feedback could be used to drive improvements and innovation into Microsoft products. The purpose of this research was to determine the ease with which people with visual disabilities, people with mobility impairments, and older adults found and used the accessibility features in Windows XP. The investigators were also interested in whether or not the participants could use the accessibility applets. Finally, the investigators examined the issues of how users self-identify and how accessible features can be provided and explained in such a way as to be perceived as useful for as wide a population as possible. The following research questions were addressed in this study:
Participants
Eight adults with visual impairments, six adults
with mobility impairments, and seven older adults participated in the
study. Activities were located at the Laboratory for Usability Testing
and Evaluation (LUTE) at the University of Washington. Individuals were
recruited from the greater Seattle community by DO-IT staff.
Demographic information on the participants can be found in Tables 1-3
at the end of this article.
Participants in the group with mobility impairments had the highest
level of computer proficiency while the group of older adults
self-reported at the lower end of the proficiency scale. Regarding
gender, all participants in the mobility-impaired group were male,
while the other two groups were more gender-balanced. Finally, all
participants in the visually impaired group used assistive
technologies, five out of six participants in the mobility-impaired
group used assistive technologies, and no members of the older adult
group used assistive technologies. It is interesting to note that
participants in the latter group expressed an interest and a need for
some of these technologies when exposed to them at the end of their
study session.
Each usability test session was approximately two hours. During each session, participants performed seven or eight basic tasks on the computer. These tasks are presented in the Results section of this article. For participants who were assistive technology users (as reported in Tables 1-3), the assistive technology was set up before the session. Data collected included videotaped observations, the time to complete each task, and answers to a brief series of questions after each task. The "think aloud" protocol, where participants are asked to verbalize their thought processes as they work, was used throughout each session.
Time on Task
Participant time scores were recorded for each task. The average time to complete each task for each group is presented in Table 4 at the end of this article. Overall, the participants with visual impairments spent 41% more time on the tasks than those with mobility impairments. While user levels of expertise varied slightly, with the group with mobility impairments having a slightly higher number of expert and advanced users, such a high level of discrepancy cannot be attributed to this factor alone.
Older adults, however, completed the tasks with an average time that was 34.5% faster than the two groups with disabilities. The notable exception was that older adults took longer to complete the "deleting files" task, which two older adults could not complete and no older adult completed to the degree of accuracy generally achieved by members of the two groups with disabilities.
Ease of Use and Satisfaction
Participants were asked in a post-activity survey to rate their satisfaction with the product in three categories: ease of use, product usefulness, and ease of finding features.
When rating the product for ease of use, users with visual impairments found the accessibility applets harder to use and harder to find than those with mobility impairments. When rating the product accessibility features for usefulness, users with visual impairments considered them more useful than participants with mobility impairments indicating a greater reliance on the features. Older adults either did not use the accessibility features or found them more difficult to locate and use, and rated these features at low levels in all three categories.
Usability Issues
Participants identified over 60 significant usability issues in Windows XP. Key problems for participants and selected examples are summarized in the following categories that usability testing, in general, highlights:
Overall, participants with visual impairments had a much more difficult time with the tasks than did participants with mobility impairments. This disparity manifested itself in various ways in the data. Individuals with visual impairments needed significantly more time to complete the tasks (all of which were fairly common, basic tasks). These participants indicated that they felt more reliant on and less satisfied with the accessibility features built into the operating system.
As expected, older adults echoed many of the issues exposed during testing with participants from the low-vision and mobility-impaired groups. Overall, the older adult participants spent much less time on the tasks than the group with mobility impairments, even though the older adults were generally less experienced than the participants with disabilities. Interestingly, however, the older adults did not rate the system as more satisfying for them. There are several possible reasons for their relatively lower satisfaction ratings.
Although they often blamed themselves for their mistakes and misunderstandings, the older adults' ratings of the system and their overall experience were more in line with the usability engineers' perception of their task performance. This included, at times, a less favorable assessment of the system than their counterparts in the first two groups. The participants with disabilities, on the other hand, may have been more willing to appreciate the software's attempts to accommodate their needs, feeling that any accommodation is better than nothing. This response may have resulted in satisfaction ratings that were higher than the usability engineers' perception of their task performance. Finally, even though there were more beginners in the older adult group, they did not have disabilities that required the use of assistive technologies to complete the tasks. Thus, they did not encounter compatibility problems that sometimes exist between the operating system and assistive technologies produced by other companies.
As with any usability test, new usability issues will always become apparent with more participants. In this case, we believe that one of the greatest contributions the older adults provided was the further validation of the findings of the earlier tests. In addition, comments from the older adults during testing helped highlight the challenge in marketing accessibility features to members of a distinct population who would benefit from the features but do not self-identify as people with disabilities.
Clearly, the need for more extensive and pervasive usability evaluations with users with disabilities is critical, especially given the reliance those with disabilities have on computers and the Internet. Still, few researchers have published studies that developers and Web designers can use to guide them. One notable exception is Coyne and Nielsen's report Beyond Alt Text: Making the Web Easy to Use for Users with Disabilities (2001). Indeed, many of our findings mirror those of Coyne and Nielsen. For example, we both found that besides being considerably easier for sighted users than for users with visual impairments, conducting tasks under Windows XP was easier for users with mobility impairments than users with visual disabilities. In addition, information that was theoretically "accessible" was not compatible with some assistive technologies (e.g., several links highlighting the accessibility features were unreadable by a screen reader).
Regarding product development, the usual recommendation is to conduct another round of testing after suggested changes to the user interface have been completed. In this case, however, the authors suggest conducting tests before that stage of product revision. If user research were conducted on competing feature ideas and prototypes of these accessibility features before code writing, the authors feel the best possible design could emerge, and for a fraction of the combined cost of initial development and later potential rework.
Reactions from Microsoft staff suggest the ultimate impact of the project was to improve usability of the software for people with disabilities. For example, one product planner with the Microsoft Accessible Technology Group stated, "This XP usability study provided our team with valuable usability feedback, as well as highlighted the importance of user research throughout the product cycle. To continue improving product design based on grounded customer data, our team has increased our emphasis on market and user research. This research must encompass the wide range of users who would benefit from assistive technology, which includes individuals with difficulties and impairments, not to mention the aging workforce and even individuals who want to customize how they interact with their computer."
The use of electronic and information technology is ubiquitous-in education, employment, and other facets of daily living. Full access for people with disabilities is important on ethical, economic, and legal grounds. However, the potential of information technology to level the playing field for people with disabilities will not be realized unless technology products are accessible to them. The usability study reported in this paper provides an example of how information technology accessibility can be addressed within a typical usability study that includes individuals with disabilities. Methods and findings from this study suggest that even though a product or product features are considered "accessible", consumers with disabilities may still have difficulty using the product easily and efficiently.
In light of these findings and previous studies, it is recommended that access issues be considered as products are being tested and developed, and accessibility and usability should both be measured in tests that include people with a broad range of disabilities, some using assistive technology. Such efforts will contribute to making products accessible to a maximum number of potential users, as well as allowing schools and other organizations to make purchases that meet legal requirements regarding accessibility. As greater numbers of individuals live to older ages, some with diminished abilities, the potential for increased market share for products that are accessible and usable is clearly in a state of growth. Our hope is that further research in this critical arena will continue to grow and add to solutions for this long-neglected population.
American with Disabilities Act of 1990, 104 STAT.327, Retrieved December 12, 2004, from http://www.usdoj.gov/crt/ada/statute.html
Burgstahler, S. (2003). Web-based distance learning and the second digital divide. M. Hricko (Ed.). Design and implementation of web-enabled teaching tools. Hershey, PA: Information Science Publishing, Idea Group Inc.
Brewer, J. (2001, January 4). How people with disabilities use the Web W3C Working Draft. Retrieved December 12, 2004, from http://www.w3.org/WAI/EO/Drafts/PWD-Use-Web/overview.html
Center for Universal Design North Carolina State University (2002) What is universal design? Retrieved December 12, 2004, from http://www.design.ncsu.edu/cud/univ_design/ud.htm
Coyne, K. P., & Nielsen, J. (2001). Beyond ALT text: Making the Web easy to use for users with disabilities. Fremont, CA: Nielsen Norman Group.
Disabilities, Opportunities, Internetworking and Technology (2004). DO-IT Snapshots 2001. Seattle: University of Washington. Retrieved December 12 2004, from http://www.washington/edu/doit/Snapshots
Gould, J. D., & Lewis, C. (1985). Designing for usability: Key principles and what designers think. Communications of the ACM, 28(3), 300-311.
Gunderson, J., & Mendelson, R. (1997). Usability of World Wide Web browsers by persons with visual impairments. Paper presented at the annual meeting of the Rehabilitation Engineering & Assistive Technology Society of North America.
Hasselbring, T. S., & Glaser, C. H. (2000). Use of computer technology to help students with special needs. Future of Children, 10(2), 102-22.
Hong, J. I., Li, F. C., Lin, J., & Landay, J. A. (2001). End-user perceptions of formal and informal representations of web sites, Adjunct Proceedings of CHI 2001, Seattle, WA, 385-386.
Kaye, H. S. (2000). Disability and the digital divide. Disability Statistics Abstract. San Francisco, CA: Disability Statistics Center, University of California, San Francisco, and Washington, DC: US Department of Education National Institute on Disability and Rehabilitation Research.
Muller, M., Matheson, L., Page, C., & Gallup, R. (1998). Methods & tools: Participatory heuristic evaluation. Interactions, 5(5), 13-18.
National Organization on Disability. (2004). 2004 National Organization on Disability/Harris survey of Americans with disabilities. Retrieved December 12, 2004, from http://www.nod.org/content.cfm?id=1537
Nielsen, J. (1996) Usability testing metrics. IEEE Software, 13(6), 12-13.
Nielsen, J., & Landauer, T., (1993). A mathematical model of the finding of usability problems. In Proceedings of ACM INTERCHI'93 Conference (pp. 206-213). Amsterdam, The Netherlands.
Office of the Federal Register, National Archives and Records Service, General Services Administration. (2000, December 21). Electronic and information technology accessibility standards. The Federal Register, 65(246), 80499-80528.
Patrick, D. L. (correspondence to Senator Tom Harkin, September 9, 1996). Retrieved December 12, 2004 from http://www.usdoj.gov/crt/foia/cltr204.txt
Rowan, M., Gregor, P., Sloan, D., & Booth, P. (2000, 14 November). Evaluating web resources for disability access. Paper presented at the fourth international ACM SIGCAPH Conference on Assistive Technologies. Washington, DC.
Schmetzke, A. (2001) Online distance education-'Anytime, anywhere' but not for everyone. Information Technology and Disabilities, 7(2). Retrieved December 12, 2004, from http://www.rit.edu/~easi/itd/itdv07n2/axel.htm
Section 508 of the Rehabilitation Act of 1973 (1998,
amended). 29 U.S.C. 794(d).
December 12, 2004, from http://www.access-board.gov/sec508/guide/act.htm
Success stories (2002). San Rafael, CA: The Alliance for Technology Access. Retrieved December 12, 2004, from http://www.ataccess.org/resources/fpic/successes.html
Taylor, H. (2000). How the Internet is improving the lives of Americans with Disabilities. The Harris Poll #30. Los Angeles; Creators Syndicate, Inc.
Thompson, T., Burgstahler, S., & Comden, D.
(2003). Research on web accessibility in higher education. Information.
Technology and
Disabilities, 9(2). Retrieved December 12, 2004, from http://www.rit.edu/~easi/itd/itdv09n2/thompson.htm.
US Department of Commerce (2000). Falling through the NET: Toward digital inclusion. A report on Americans' access to technology tools. Retrieved December 12, 2004, from http://search.ntia.doc.gov/pdf/fttn00.pdf
Virzi, R. (1992). Refining the test phase of usability evaluation: how many subjects is enough? Human Factors, 34(4), 457-468.
This study was a collaboration between the University of Washington and Microsoft. We would like to thank Microsoft for its contribution.
Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
Demographic Data for Participants with Visual Impairments
| Age | Gender | Computer Proficiency | Disability | Assistive Technology |
| 42 |
F |
INT | Low vision |
ZoomText |
| 50 | M | ADV | Low vision | Large monitor, high contrast, large fonts |
| 39 | M | INT | Blindness | JAWS |
| 56 | F | INT | Low vision, slow focusing | Dragon Naturally Speaking, Velcro on keyboard |
| 43 | M | INT | Low vision | ZoomText, large monitor, Dvorak keyboard |
| 39 | M | ADV | Low vision | JAWS, DecTalk, ZoomText |
| 54 | F | INT | Blindness | JAWS, hot keys |
| 27 | M | ADV | Blindness | JAWS, DecTalk |
Participants reported their computer proficiency as beginner (BEG), intermediate (INT), advanced (ADV), and expert (EXP).
Demographic Data for Participants with Mobility Impairments
| Age | Gender | Computer Proficiency | Disability | Assistive Technology |
| 27 | M | EXP | Cerebral Palsy | Headwand, sticky keys, trackball |
| 33 | M | INT | Cerebral Palsy | None |
| 23 | M | ADV | C 5/6 Quadriplegia | Hand splints, raised computer table |
| 23 | M | ADV | C 5/6 Quadriplegia | Hand splints, raised computer table |
| 22 | M | INT/ADV | C 4/5 Quadriplegia | Adap2U, Adapoint, on-screen keyboard |
| 30 | M | INT | C 4/5 Quadriplegia | Trackball, sticky keys |
Participants reported their computer proficiency as beginner (BEG), intermediate (INT), advanced (ADV), and expert (EXP).
Demographic Data for Older Adult Participants
| Age | Gender | Computer Proficiency | Disability | Assistive Technology |
| 59 | M | BEG | None | n/a |
| 69 | M | BEG | None | n/a |
| 58 | M | BEG | None | n/a |
| 62 | F | BEG | None | n/a |
| 52 | F | BEG | Apperception visual anoxia/Ocularmotor dysfunction | None |
| 49 | M | BEG | Hand tremors | None |
| 78 | F | BEG | None | n/a |
Participants reported their computer proficiency as beginner (BEG), intermediate (INT), advanced (ADV), and expert (EXP).
Average Time on Task for Each Group in Minutes
| Task | Visual Impairments | Mobility Impairments | Older Adults |
| Set up the computer | 21:00 | 18:30 | 6:51 |
| Navigate the Web | 11:00 | 10:00 | 5:09 |
| Write and save a letter | 10:00 | 6:00 | 3:37 |
| Edit a letter | 3:30 | 3:30 | 2:29 |
| Copy a file | 5:40 | 2:30 | 3:33 |
| Delete multiple files | 8:00 | 5:20 | 8:09 |
| Create a new account | 10:00 | 6:00 | 3:37 |