NTID


Vol. 3 No. 1, Winter 1998


Deaf children exposed to sign language and hearing children exposed to spoken language from an early age have very simiar language models, thus accounting for the fact that they have very similar courses of language development. For a discussion of this conclusion, see Deaf Children's Language Abilitities as Reflected in Writing and Signing below.

On the Frontier of New Directions

Robert Davila, Vice President of NTID


In this newsletter, NTID Professor Ross Stuckless reports on the 1997 symposium on automatic speech recognition (ASR). As I remarked at that symposium, the concept of translating speech to print is not new to the deaf community. More than 100 years ago, an editorial in the American Annals of the Deaf described a newly invented mechanical device, the glossograph, which was to be inserted in the speaker's mouth to transcribe spoken words; Edward Allen Fay (1883) stated, "Every invention in this direction is of interest to our readers as holding out the prospect of a great possible boon for the deaf."

Given the rapid advances in communication technology over the past 25 years, we can visualize an ASR system that benefits deaf and hard-of-hearing people. More importantly, NTID, in association with the other colleges of RIT, is well positioned to play a major role in advancing innovative communication technologies for deaf and hard-of-hearing people. Indeed, NTID's involvement in this work is appropriate, necessary, and should always be a staple of our research and development efforts. Truly, which institution is a better candidate than NTID to pioneer new technology for deaf and hard-of-hearing people?

For NTID to remain on the cutting edge of new technological developments, however, we must establish partnerships with private industry and government; these collaborative efforts would ensure that the needs of deaf and hard-of-hearing persons are considered at the funding and design stages of new products. In our role as a partner with industry and government, we must serve as a powerful advocate for deaf and hard-of-hearing people.

To paraphrase an old maxim, necessity drives the engine of technological innovation. For example, the telephone, invented by teacher of the deaf Alexander Graham Bell, remained inaccessible for profoundly deaf per sons for almost a century following its invention until a deaf scientist, Dr. Robert H. Weitbrecht, invented the acoustic coupler that enabled transmission and reception of teletypewriter (TTY) signals over the telephone line. This invention was no accident. Weitbrecht created one of history's most valuable technological innovations and communication breakthroughs for deaf people by actively seeking to overcome the inability of this group of people to use the telephone. We need more scientists and researchers who know what deaf and hard-of-hearing people need and who go about finding ways to meet that need.

The challenge for NTID is to pick up Dr. Weitbrecht's torch. NTID's knowledge and understanding of deaf people and deafness, coupled with RIT's vast resources, make this university community the logical place to launch a program of research and development designed to create new technologies and new opportunities for those whom we serve--deaf and hard-of-hearing people.

I look forward to a new direction in technological innovation with enthusiasm and optimism.


Robert R. Davila, Vice President for NTID


Deaf Children's Language Abilities as Reflected in Writing and Signing
by Marc Marschark

Marc Marschark is a research faculty member at NTID. He has published six books and more than 70 articles, and is founder and editor of both the Journal of Deaf Studies and Deaf Education and the Counterpoints: Cognition, Language, and Memory book series, both for Oxford University Press. For more information, contact Marschark at MEMRTL@rit.edu


To what extent can deaf children's language productions be taken as indicative of their underlying language skills and cognitive abilities? Intertwined here are issues of creativity, writing ability, how we teach deaf children, and the role of sign language in that education as well as the expectations placed on those children by their teachers and parents.

Despite more than twenty years of studying the language of deaf children, we know very little about what deaf children say. We do know a lot about the character of their English writing skills and the structure of their sign language. But these studies have been initiated primarily by investigators interested in linguistics. My interests lie more in the psychology of language: I want to understand why, when, and how children use language and how their language reflects what they know. I don't really care about their knowledge of grammar or the size of their vocabularies.

Reflecting on this focus, I have long been interested in the stories that deaf children tell and the creativity and knowledge of the world reflected in those stories. I am constantly surprised at how little attention is given to this issue in textbooks intended for future teachers of deaf children, and how few educators recog nize the role of verbal creativity in fostering academic and intellectual growth. Instead, we find that conclusions about deaf children's language creativity (or, more commonly, their lack of it) are usually based on observations of their skills with written English. Although the need for literacy is a laudable goal for an educational system that seeks quality education for deaf children, I believe that drawing conclusions about their cognitive and language abilities on the basis of their performance in English can only underestimate those abilities. Assessment of deaf children's language ability is a somewhat different enterprise from assessment of their capabilities within any particular language. For many deaf children, that means we need to consider their comprehension and production capabilities in sign language as well as English.

My own research in this area started more than ten years ago with curiosity about whether deaf children use nonliteral, figurative language in their signing. At that time, the artistry and creativity of American Sign Language (ASL), at least when in the hands of deaf adults, was already well-established. Research on hearing children's figurative language abilities was also quite popular, because such skills had long been taken as an indicator of underlying cognitive skill. Studies involving deaf children, howver, uniformly have evaluated only their ability to understand English idioms, metaphors, and similes (e.g., "she worked her heart out," or "don't look a gift horse in the mouth"). Not surprisingly, given deaf children's relative inexperience with English, they typically performed poorly on such tasks, frequently leading to the conclusion that they are concrete and literal thinkers.

When we started looking at deaf children's productions in sign language, however, the situation turned out to be quite different. In several studies conducted in collaboration with Victoria Everhart, Sue West, and others, we found that when signing made-up stories, deaf children produced just as many metaphors, similes, and idiomatic expressions as their hearing peers produced in spoken stories. In some categories of nonliteral expression, they appeared even more flexible and creative than their hearing peers (Everhart and Marschark, 1988). In one study, we compared deaf and hearing children in the extent to which they used figurative language in the production of written stories and in the production of signed and spoken stories (for deaf and hearing children, respectively). Consistent with our previous studies, we asked children to tell us about what would happen if they were picked up by a UFO and taken to another planet or if they discovered a "lost city" under the ocean. In this study, the children ranged in age from seven to 14 years.



As can be seen in Figure 1, hearing children produced signifi cantly more creative constructions in their writing than did deaf children--not a surprising finding given the emphasis of most educators of deaf children on the use of correct, literal English and the infrequency of exposure of deaf children to creative English models. Figure 2, in contrast, shows that the same deaf children were at least comparable to their hearing peers in their use of creative language devices when they were signing their stories and the hearing students were speaking their stories. These results clearly demonstrate that previous studies indicating that deaf children were concrete and literal in their language were biased by the use of written English assessments.



Such biases can be found in other kinds of studies as well. For example, researchers and English teachers have long debated why deaf children do not seem to understand the underlying structure of stories they read (who did what to whom? how? and why?) and fail to organize their writing around such structures. Some investigators have claimed that because most deaf children--those with hearing parents--are not exposed to normal day-to-day conversations in the home, they do not learn discourse rules as part of their language repertoires. Other investigators have claimed that the failure to apply discourse rules is only a function of deaf children's unfamiliarity with the rules of written English. Our reasoning went like this: Quite simply, if deaf children really do not understand discourse rules, it should be evident in their signing as well as their writing.

Marschark, Mouradian, & Halas (1994) explored this issue in two experiments. In the first experiment, we compared stories that were signed by deaf students and spoken by hearing students, all aged eight to 16 year. In the second experiment, we compared stories written by the same students. Analyses of the meaning structures contained in signed and spoken stories indicated that the two groups were esentially identical in all respects. This result was not unexpected given our experience with the language creativity and flexibility of deaf children, although it went against the strong position that deaf children do not understand discourse rules at all. We were more surprised, however, by the results obtained with written stories. Although the written English of the deaf students' stories was not as good as that of the hearing students' stories in terms of grammar of vocabulary, the underlying conceptual structures were fully comparable in the two groups. Both aspects of this situation can be seen in the short story shown in Figure 3, written by a deaf 11-year old. The story is clear and well structured in terms of its content, but its English leaves much to be desired.

Figure 3.
Writing Sample


Stories like this from many students in our study thus indicated that at least from eight years of age onward, deaf and hearing children's written stories are equal in the complexity of their meanings and in their conceptual well-formedness, even if they differ in the quality of their English. The problem, of course, is that teachers and parents often cannot get past the poorly-phrased English of deaf students' writing to see that a story really is well constructed and creative. Rather than rein forcing children in such situations and encouraging them to write more, there is often a tendency to be discouraging and thus to make writing seem like a chore.

Using methods like those described above, we recently have been exploring the language directed to deaf children by deaf moth ers, the language directed to hearing children by hearing mothers, and that directed to both deaf and hearing children by hearing (bilingual) mothers who had ASL as their first language (see also Marschark, Everhart, & Dempsey, 1991). These studies have been important in providing information about the language models available to young deaf children and in allowing us to examine changes in the nonliteral content and meaning structures in the stories produced by deaf individuals from the early school years to adulthood. Overall, those studies have shown that hearing mothers and deaf mothers are very similar in the stories they tell young children in terms of both figurative versus nonliteral content and their underlying conceptual structures, regardless of whether they use sign language or spoken language. There are differences in the way that some story elements are expressed in the two modes, but these pale in comparison to the similarities. Most importantly, these results converge with other findings in demonstrating that deaf children exposed to sign language and hearing children exposed to spoken language from an early age have very similar language models, thus accounting for the fact that they have very similar courses of language development.

Taken together, the studies described here consistently lead to two conclusions. One is that evaluation of deaf children's language abilities, or assumptions about the cognitive abilities underlying their language productions, require more than can be obtained through tests involving the comprehension or production of English. The second conclusion is that the language to which deaf children are exposed will affect the language that they use and the ways in which they interpret what goes on around them. Neither of these conclusions seems very surprising or very radical. Nevertheless, in many venues, deaf students are still confronted with educational practices that funnel them into prescriptive learning of English rather than providing them with opportunities to explore and play with language, in ways that we recognize are important for gaining literacy. Such practices have been tried, without success, for over 100 years. Maybe it's time to try something else.

As I indicated in an earlier NTID Research Bulletin editorial (Volume 1, No. 2), it is time for a stronger link to be made between research and teaching. Both areas need to better inform each other and maintain an active dialogue. To do anything else will only waste time and resources...and the educational opportuni ties of deaf students throughout the country.

References



Special Applications of Automatic Speech Recognition (ASR) with Deaf and Hard-of-Hearing People: Part II
by Ross Stuckless

A member of the senior research faculty at NTID, Ross Stuckless began his career more than 40 years ago as an English teacher at the American School for the Deaf in Hartford, Connecticut, where his brother David was among his students. Ross is optimistic that future applications of automatic speech recognition will contribute substantially to the quality of life among deaf children and adults, and others who share their lives. For more information on ASR, contact Stuckless at ERSNVD@RIT.EDU


1997 Symposium on ASR
I once heard a deaf person say wistfully that he longed for a "little black box" he could carry around in his pocket to enable him to become more independent in his communication with hearing people. To date, widespread use of automatic speech recognition (ASR) in classrooms and other group settings in which deaf or hard-of-hearing people are participants, as discussed in Part I of this article (NTID Research Bulletin, 2(3), 1997), has been constrained by the need for a third-party operator. However, in 1997 we came a step closer to that "little black box" when two new ASR products came on the market, both capable of processing large vocabulary, speaker-adaptive, continuous speech.

Fortuitously, announcements of Dragon Systems' NaturallySpeaking, Personal Edition and IBM's ViaVoice Gold were announced at the same time that the University of Rochester and the Rochester Institute of Technology hosted the Frank W. Lovejoy Symposium on Applications of Automatic Speech Recognition with Deaf and Hard of Hearing People in April, 1997. Most of what follows in this article is based on observations from, and presentations and formal discussions during, that symposium.

Complexity of recognizing natural language
The first commercial applications of ASR were directed toward a more controlled form of language, and not toward the more free-flowing form we commonly use in spoken conversation. Naturally-spoken language is much more difficult to recognize automatically than the formal, carefully organized language we use in dictating or reading aloud from text, resulting in a considerably higher error rate for the former. In his presentation, Michael Picheny (1997) talked about dysfluencies common to spontaneous speech--the "um's," "ah's" and "you know's," the false starts and the restarts, all of which complicate the task of speech recognition.

However, as we move beyond the task of recognizing discrete words one at a time, we cannot depend on their acoustic characteristics alone. Harry Levitt (1997) indicated that while acoustic cues are important, they are not sufficient for accurate automatic recognition of naturally-spoken language and must be coupled in clever statistical/computational ways with linguistic cues. James Allen (1997) carried this point further. He distinguished among syntax, semantics, and pragmatics, all of which provide important cues, particularly in the recognition of spontaneously-spoken language, drawing close to what we regard as artificial intelligence.

Another practical problem associated with efforts to record conversational and other informal forms of speech is that ASR can transcribe only what is spoken. No punctuation, new sentence, new paragraph, or other visual markers accompany the automatically-recognized text unless the speaker issues a voice command (or keystroke), such as "period" or "new paragraph," to instruct the computer to do so. Otherwise, the text would be displayed as an unbroken string of words.

Early generation continuous speech recognition

Mark Mandel (1977) of Dragon Systems extemporaneously demonstrated the pre-release alpha version of his company's first generation continuous speech product, NaturallySpeaking, Personal Edition. The transcript appears to the right without correction (errors are underlined). The speaker added punctuation by voice as he spoke. Four errors appeared in this 129 word continuous speech production, making it better than 95% error-free and quite readable. A second version, NaturallySpeaking Deluxe, has since come on the market with several refinements.

IBM has also released a second version of its continuous speech recognition product, ViaVoice Gold. I have used both products and both performed very well in a dictation or text-reading mode. I also used them in both lecture and conversational modes, where their performance was unsatisfactory. I do believe that with some speaker adaptations and the addition of an equipment accessory or two, performance could be considerably improved. However, whether it would be satisfactory would depend on what Kathryn Woodcock (1997) identified as an ergonomic model of the system's requirements, which combines the user and the device within an environment, performing a task.

 Fellow friends and colleagues, As ec Well I promised you this was a buggy version. I try to keep my promises even when it embarrasses me.
   As you see, this is not ready for shipment as a product, But it does fairly well at recognizing continuous speech. So well that we're confident that we will have a product out in a timeframe we mentioned.
   In order to use this--well--in order for a deaf person to use this in conversation with a hear ing person, the hearing person would have to train this product. This is not speaker independent yet. It could be but I'm not promising that. But as you see! it does pretty well at recognition and even with correction gets a great deal of the information across.

What's next in speech recognition?
Michael Picheny (1997), an expert in ASR research, suggested that "the biggest research challenge over the next couple of years will be to come up with models for handling rapid conversational speech." He also talked about other priorities, including the need to deal with recognition problems posed by dysfluencies and accents in speech, background noise, and telephone characteristics, e.g., narrow bandwidth.

These challenges notwithstanding, Picheny made several personal predictions, qualifying them by saying that their enabling systems "may have lots of errors, but technology should allow us to begin using them within the following timeframe." He felt that by the end of 1998, ASR systems should be capable of handling speech over the telephone at 20% error rates. In 1999, limited ASR systems capable of transcribing broadcast news (though perhaps not TV shows) should be available. By the year 2001, we may see ASR systems that transcribe meetings. There may also be hand-held devices (possibly in communication with more powerful remote ASR systems) that people can carry around in their hands to read transcriptions of actual ongoing conversations and events.

Personal thoughts from the symposium
Like others who participated in the symposium and/or have read its proceedings, I came away with new information and expectations. First, we need to encourage major ASR developers to recognize deaf and hard-of- hearing people as members of a potential niche market. We should also take the initiative ourselves in adapting new systems and devices as needed.

Second, the potential value of ASR in telephone communication involving deaf and hard-of-hearing people is quite significant, and efforts in this direction should be pursued vigorously. Third, as Kathryn Woodcock (1997) suggested, we should be attentive to the interaction of user, device, environment, and task in targeting ASR applications.

And, finally, we need to understand that transcription errors in ASR cannot be eliminated in the foreseeable future. Nonetheless, we need to remember that deaf and hard-of-hearing people will not accept and use ASR-based systems and devices that do not meet their standards for accurate and reliable communication.

Conclusion
If I had a disappointment, it was that we had no time to delve more deeply into some topics and to open others. As a former teacher of deaf children, I wanted discussion about ASR's potential for English language learning, both at home and in school. As an educational researcher, I wanted discussion about how ASR might assist deaf and hard-of-hearing students in mainstreamed classes. And as a faculty member in a career-oriented college and university, I wanted to explore some thoughts about how ASR could be adapted to communication needs in the workplace. Another time...

References

To obtain a copy of the proceedings from the Frank W. Lovejoy Symposium on Applications of Automatic Speech Recognition with Deaf and Hard of Hearing People, contact Ross Stuckless at ERSNVD@RIT.EDU and type "ASR Proceedings" on the subject line. Also, you can review and download the proceedings in their entirety at http://www.isc.rit.edu/~ewcncp/Lovejoy.html


NOTES OF NOTE

An article titled "Round-off error, blind faith, and the powers that be: A caution on numerical error in coefficients for polynomial curves fit to psychophysical data" by Vince Samar and Carol De Filippo of NTID's Department of Applied Language and Cognition Research, will be published in the April 1998 issue of the Journal of Outcome Measurement. The article was written to focus awareness within the research community on a deceptively simple methodological pitfall associated with the use of curvefitting software and procedures to obtain quantitative measurements from psychophysical data sets. Written in a light tone, the article describes an anecdotal experience with a real data analysis problem that made the authors stop and think twice about their faith in the analytic products of such software. For more information, contact Samar at e-mail VJSNCR@RIT.EDU

Harry Lang and John Albertini, Department of Educational and Career Research, have been invited to present a two-part workshop at the Model Secondary School for the Deaf (MSSD) in Washington, DC. The workshop will apply concepts they developed in the Access to English and Science Outreach Project (AESOP) to assist the MSSD faculty in designing their integrative curricula. Between the two workshops, the teachers will experiment with strategies learned during the first workshop and evaluate their effectiveness. (see NTID Research Bulletin, 2(3), 1997, for a discussion of AESOP.) For more information, e-mail Lang at HGL9008@RIT.EDU

Two articles titled "Effect of Sign Task on Speech Timing in Simultaneous Communication" and "Production and Perception of Final Consonant Voicing in Speech During Simultaneous Communication" were published in the November/December issue of the Journal of Communication Disorders. These articles were co-authored by Robert Whitehead, Department of Applied Language and Cognition Research, Brenda Whitehead, Department of Speech/Language, and Dale Metz and Nicholas Schiavetti of the State University of New York at Geneseo. For more information, contact Whitehead at RWWNCR@RIT.EDU

Deaf and hard-of-hearing high school students who apply to RIT through NTID will receive an interactive viewbook on CD-ROM that runs on Macintosh, Windows 3.1, and Windows 95/NT platforms. The first of its kind for NTID, the CD showcases QuickTime® movies of 14 current students and presents a walking tour of the RIT campus, information about majors, student support services, sports and social activities, and visitor information, including travel directions and a pictorial tour of Rochester. Users are greeted by Dr. Robert Davila, Vice President for NTID, and student guides who introduce each of the eight sections of the CD. For more information, contact NTID's Department of Recruitment and Admissions, 716/475-6700 (V/TTY), WWW: http://www.rit.edu/NTID/CO/ADMIS

Vince Samar, Ila Parasnis, and Jerry Berent, Department of Applied Language and Cognition Research at NTID, have a chapter which will appear this spring in Psychological perspectives on deafness, vol. II, published by Lawrence Erlbaum. "Learning disabilities, attention deficit disorders, and deafness" reviews the literature on learning disabilities (LD) and attention deficit disorders (ADD) in the deaf and hard-of-hearing population within the broader context of mainstream research on LD and ADD. Some promising new directions for research on evaluation and remediation of LD and ADD are identified. For more information, contact Samar at e-mail VJSNCR@RIT.EDU

Insert:
IMPLICATIONS OF NTID RESEARCH FOR DEAF AND HARD-OF-HEARING PEOPLE

NTID Research Bulletin
The NTID Research Bulletin is published periodically by the Center for Research, Teaching and Learning, National Technical Institute for the Deaf, a college of Rochester Institute of Technology. It is available without charge.

Opinions expressed in the NTID Research Bulletin do not reflect those of NTID or RIT. Your comments, questions, and requests for more information are welcome. See following address.

If you wish a copy of the NTID Papers & Publications 1996 or if you know of colleagues who would enjoy receiving the NTID Research Bulletin, please send names and addresses to:

NTID Research Bulletin, Bldg. 60-2238
52 Lomb Memorial Drive
Rochester, NY 14623-5604
Fax: 716-475-6500, E-mail: ASKCRTL@RIT.EDU

Marc Marschark, Director, CRTL
Gail Kovalik, Editor

Center for Research, Teaching and Learning
National Technical Institute for the Deaf

Return to CRTL Home Page