C-Print captionist Joyce Gambacurta provides real-time transcription services for RIT student Don Carter in a Science, Technology and Values class, taught by RIT professor Robert Paradowski. After class, Don and other deaf and hard-of-hearing cross-registered students receive a printed copy of the transcript.
Educational Research and Research Education
On a trip recently, I met with a colleague who was conducting a
research project involving deaf children at a local school. In
discussing his study, my colleague expressed surprise at how poorly
the children were performing, clearly suggesting lags in cognitive
development. The odd thing about this was that the task he was
using is similar to one that I have used myself, and the children
I tested did just fine.
Could the discrepancy be due to some difference in the curriculum of one school or the other? Could it have resulted from one of the seemingly minor differences between our studies? To try to find out, we went to the school, chatted with students, and tried my version of the task informally with two of them. To my colleague's surprise, both children grasped the task immediately and did fairly well. What does it all mean?
Because my colleague does not know how to sign, he was using certified interpreters as research assistants. In administering the task, they did exactly what they were instructed to do. They were even kept in the dark about the purpose of the study, so as to avoid testing bias. Aha! As it turns out, this colleague really did not know much about deaf children. He naively assumed that the method would work exactly as it had in two earlier studies with hearing children, and that outcomes would mean the same thing. The only problem was that the task did not make any sense to the deaf children!
If my colleague had understood sign language and/or understood deaf children, he would have known that the task was structured wrong. Or, if the interpreters/assistants had backgrounds in research or had the study appropriately explained to them, they would have realized that something was amiss. The scary thing is that this research would have revealed significant differences between deaf and hearing children in a domain where they really are quite comparable. Worse, the study might well have been published and accepted by teachers and other investigators around the world.
The fact that this task had to be structured slightly differently to get similar performances from deaf and hearing children is an interesting research question in its own right. However, it clearly would have been wrong to conclude that these deaf children lacked the necessary cognitive skill to perform the task. While such errors may not happen often, the event serves as a potent reminder that we should always be sure that we are testing what we think we're testing. More broadly, it should be a reminder that it is practically impossible to do (good) research with a group when you don't know them and you don't know their language.
So, when you learn of someone conducting a study with a group with whom you have knowledge and background, offer your (nonjudgmental) support. The field will benefit from your time and effort. Remember: researchers are educators, too.
Marc Marschark
Director, CRTL
C-Print: Where Have We Been?
What Did We Learn?
Where Are We Going?
by Barbara McKee, Michael Stinson, Pam Giles
Barb McKee is an associate professor in the Department of Educational and Career Research at NTID. Her research interests include the measurement of student perceptions of courses and instructors, the impact of college on students, the characteristics of successful college students, and the formative and summative evaluation of courses and programs. For more information, she can be contacted at BGMERD@RIT.EDU.
Mike Stinson is a professor in NTID's Department of Educational and Career Research. His research interests include the instruction of deaf and hard-of- hearing students in mainstream settings and the effects of technology, interpreting, notetaking, and tutoring. He is also interested in the social integration of deaf students who are mainstreamed and in the motivation of students in the classroom. In addition to research, he teaches in the Master of Science in Secondary Education of Students Who Are Deaf or Hard-of-Hearing program, which prepares teachers of the deaf. For more information, he can be reached at MSSERD@RIT.EDU.
Pam Giles is the C-Print Captionist/Training Coordinator. She has worked in various capacities at NTID for 22 years. She is an experienced C-Print Captionist/Training Coordinator, and also coordinates the day-to-day operation of the C-Print project. For more information, she can be reached at PGGNCP@RIT.EDU.
Providing for adequate communication for the deaf or hard-of- hearing student in the mainstream classroom is a complex and challenging task. A reasonable approach is support services best tailored to the individual student's needs, within constraints such as cost and availability. One such form of support, developed by researchers at NTID, is a computer-aided system for transcribing speech to print, called the "NTID Computer-Aided Transcription System," or "C-Print."
Description of C-Print
The C-Print system involves a hearing captionist (transcriber)
typing the words of the teacher and other students as they are
being spoken. The system provides a real-time text display that the
deaf student can read on a second laptop computer or a TV monitor
to understand what is happening in the classroom. In addition, the
text file is stored in the computer and can be edited, printed, and
distributed to students, tutors, and instructors after class.
Due to the speed of speech normally used by college instructors (approximately 150 words per minute), the system cannot provide word-for-word transcription. Therefore, C-Print uses two strategies to achieve the goal of including as much of the relevant information as possible: a computerized abbreviation system to reduce keystrokes, and text condensing strategies. The system employs ordinary word processing software which is augmented with a computerized abbreviation system to substantially reduce keystrokes. A set of phonetically-based rules for abbreviating words has been developed, as well as procedures for training captionists in using these rules. To deal with rapidly spoken lectures, the project developed text "condensing" or reduction strategies. The goal of these strategies is to reduce the number of words and abbreviations typed, while preserving meaning and keeping the message displayed as near verbatim as possible.
Where Have We Been?
Researchers at NTID have been working with speech-to-print
transcription systems for over 15 years. Much of the early
research was conducted with stenographic systems (Real Time Graphic
Display) by Ross Stuckless and Michael Stinson. That early work
documented the value of such systems as a classroom support service
and was a primary impetus for the development of a less expensive,
more portable system with a shorter training time for the
transcribers.
C-Print really began to "take off" when NTID received a three year Federal grant in December of 1993, which allowed the department to hire a full-time research assistant and a full-time captionist/trainer plus part-time transcribers. The primary goal of this grant was to develop the system. That goal has been accomplished.
The C-Print phonetic rules used to abbreviate words were modified significantly from what was in place at the beginning of the grant. The instructional materials used to train transcribers on the rules also underwent major modifications. Strategies to condense or summarize a lecture and accompanying training materials were developed and the entire system was piloted in a variety of RIT classrooms.
This past summer, eight people were trained to become C-Print transcribers. The trainees came from across the country (two from California, two from New Mexico, and one each from Ohio, Pennsylvania, Oklahoma and Arkansas). Data from the summer workshop is still being analyzed, but all indications are that the two week training period allowed the participants to learn the basics of both the abbreviation rules and the condensing strategies. The trainees will now have to practice with the system in order to bring their speed up to acceptable standards.
Joyce Gambacurta is currently employed as a C-Print captionist at NTID/RIT.
What Have We Learned?
* We learned that students like the system and feel the
completeness of the information is one of the major benefits.
In a questionnaire/interview study conducted from winter, 1994 through spring of 1996, results indicated that students felt they were getting more complete information from C-Print than from other sources and that this facilitated comprehension of the classroom discourse. The following comment shows how one student developed a new appreciation of the complexity and richness of the classroom dialogue with C-Print.
Interviewer:
Do you have some opinions or faults or comments about C- Print that you would like to add?Informant:
Well, I would say that it helps a lot. And it surprised me, because I never realized how much information was provided in class. Before, I always thought that the teacher did not provide enough information and it was boring; but when I was using the C-Print, it seemed more interesting. It makes me feel like I have been missing something in the past. Like I missed the last few years.Another student commented:
When I watch the interpreter and the teacher, I know that the interpreter is changing what the teacher is saying a lot, and I don't like that because I feel I am losing a lot. Most of the time I will ignore the interpreter and pay attention to the teacher. Some interpreters I have had a few times, and I know if they are good or not. So it depends on the interpreter.From the questionnaire data, almost all students reported feeling that the summarization done by the C-Print captionist was acceptable and that they were getting the important points of the lecture; only one student felt that the summarization was not acceptable. A student responded to an interview question about the extent that the captionist summarized information as follows:
Yes, I accept that it is summarized. I can hardly tell if it is summarized. It looks like she is just typing every single word that the teacher is saying. I can hardly tell that she is summarizing. When I look at the interpreter, I can tell that they are summarizing. So I can see the difference.* We learned that we can capture two-thirds of all the information provided in a classroom and three-fourths of the important information.
Information collected to date indicate that C-Print captioners record approximately 66% of the information spoken in class and 76% of the important information (Everhart, Stinson, McKee, Henderson, & Giles, 1995).
* We learned a lot about the skills and training required to become a C-Print captionist (or operator as we call them).
We have now developed several screening tests for potential captionists. These tests include a typing test, general English language ability test and a phonetic test that assesses the ability to "hear" the sounds in words--for example to hear that "c" in English is often pronounced "k"--so coffee sounds like "kfe." We have learned that it requires approximately four weeks of part-time training to bring a 60 word per minute typist with appropriate English and phonetic skills up to the level of what we have termed "apprentice" captionist. It then requires approximately one-quarter of in-class experience before an operator is ready to function without assistance.
* We have learned that it is often the "peripheral" issues that cause more problems than the actual implementation of the system.
Issues such as equipment failure, finding backup for captionists who are ill, and finding appropriate places to sit in some classrooms are some of the difficulties the project has encountered.
* Finally, we have learned that we need to know more about integrating the C-Print system into existing support services programs and we still have a lot to learn about using the system in environments other than RIT. Which brings us to the reason we applied for another grant.
Where Are We Going?
In December of 1966 NTID will receive additional funding from
the U.S. Department of Education to continue the C-Print project.
The first grant focused on the development of the system; the
second grant will focus on the implementation of the system in
different environments. C-Print will be piloted in four (possibly
five) settings where students are mainstreamed: a large,
centralized high school regional program; a program which provides
itinerant teachers and support services to deaf and hard of hearing
students attending local high schools; the college program at NTID
which has a large number of students and extensive support
services; and a group of college programs with few deaf and hard of
hearing students and limited support services.
These four types of programs approximate the variety of settings in which students are mainstreamed and will provide the opportunity to explore the potential of C-Print in each situation (Kluwin & Stinson, 1993; Stinson & Walter, 1992). The regional high school program sites will be in Irvine and San Diego, California. The other sites will be in the Rochester, New York, area. At each site the C-Print system will be provided as a support service in classes, and much of the data collection is designed to tap the experiences of student and teacher consumers.
We anticipate learning a great deal about the type of mainstream environments where C-Print can be used most effectively and the characteristics of students for which the service is the most appropriate.
C-Print is not a panacea for overcoming communication barriers faced by deaf and hard-of-hearing students. No single channel of receptive communication (e.g., speechreading, sign reception, reading) can be entirely suitable for all these students under all conditions. Evidence is accumulating, however, which indicates that a transcription system such as C-Print is an effective way of increasing accessibility to information in the classroom.
References
This research is supported by Grant H180J30011 from the U.S. Department of Education, Office of Special Education & Rehabilitative Services to Michael S. Stinson and Barbara G. McKee.
Vince Samar is an associate professor in the Department of Applied Language and Cognition Research at NTID. Samar's current research interests include assessment of language and other cognitive skills and the origins and identification of attention deficits and learning disabilities in the deaf population. He has published numerous research articles in the area of deafness, including articles on the assessment of language skills, the neural and cognitive mechanisms of language and spatial skills, the control of visual attention, and applications of captioning technology in the classroom. For more information, he can be reached at VJSNCR@RIT.EDU.
Generally, it is wrong to assume that writing tests validated on the hearing population are automatically valid for deaf or hard-of- hearing students. Language and testing factors can invalidate writing tests borrowed from programs for hearing students. Therefore, the validity of candidate writing tests must be studied before adopting them for use with deaf or hard-of-hearing students. My colleagues and I at NTID and the University of Rochester have recently published some research that addressed the problem of how to validly and fairly assess deaf and hard-of-hearing college students'overall writing skills (Berent, Samar, Kelly, Berent, Bochner, Albertini, & Sacken, 1996). The central issue concerning us was whether currently popular "objective" tests of writing, known as "indirect writing tests" and used with the hearing population in many college settings, are valid for use with a deaf and hard-of-hearing college population.
Indirect writing tests are typically multiple-choice tests of specific kinds of language knowledge such as grammar. Curiously, they do not require any actual writing on the part of students, but are generally regarded as valid for the hearing population because they measure certain component skills that seem important for good writing and they have been shown to partially predict performance in writing composition courses for that population. Indirect tests are valued primarily because they are easy to administer and score.
"Direct writing tests" work differently. A student writes an essay on a given topic and then a panel of readers rates the quality of that student's writing on a scale from completely incompetent writing to completely competent writing. Direct tests are primarily valued because they have clear face validity (i.e., they apparently test what they say they're testing) and are reliable for nearly any population when appropriate attention is paid to details of administration, testing conditions, population factors, and scoring (Huot, 1990).
For the deaf and hard-of-hearing population, indirect tests pose several potential pitfalls to accurate writing assessment. In particular, indirect tests may be biased toward aspects of grammar, mechanics, vocabularly, or world knowledge that are problematic for some students or are culturally atypical. Such factors may not seriously compromise the quality of many students' actual spontaneous writing. On the other hand, indirect tests often fail to sample other significant aspects of written language, such as organization, content, and coherence, skills in which some deaf and hard-of-hearing students might excel and which might have greater influence on the quality of their spontaneous writing. Additionally, indirect tests confound reading skills with writing skills. This last point is important because some students, especially some deaf and hard-of-hearing students, might be able to express themselves well in their own words, but might not be as facile in reading the specific content, grammatical structures, and vocabulary of a given set of test questions. Furthermore, some students might be dyslexic without having comparable difficulty writing. Therefore, this confound might introduce serious error in writing assessment for some deaf and hard-of-hearing students.
These considerations led us to compare the performance of 56 deaf and hard-of-hearing college students at NTID and RIT on two potentially appropriate indirect writing tests and on a direct measure of the quality of actual writing samples. We carried out an exhaustive search for published, standardized indirect measures of writing competency with potential for use with deaf and hard-of- hearing college students, and finally selected the Written English Expression Placement Test (Educational Testing Service, 1985) and the New Jersey High School Proficiency Test: Writing Section (Cooperman & Bloom, 1988) for further study.
We administered these tests to the 56 students, along with a direct writing test, namely the Test of Written English or TWE (Educational Testing Service, 1986). The TWE is a highly reliable subtest of the Test of English as a Foreign Language. It allows students to generate and organize ideas on paper, to support those ideas with examples or evidence, and to use the conventions of standard written English. The TWE is scored by readers who rate the proficiency of the essays on a six point scale.
To analyze our results, we determined how well the indirect writing tests were able to predict students' scores on the direct measure, the TWE. If the prediction were strong, then the indirect tests might be safely used to assess writing skills in the deaf and hard-of-hearing college population. Our results were first adjusted for the imperfect reliability of both the direct and the indirect tests. This adjustment removed statistically the influence of unknown and irrelevant factors that can introduce measurement error into test results such as fatigue, boredom, health factors, unintended response errors, environmental distractions, and other specific factors that could interfere with students' abilities to deal with the test material.
Under the conservative assumption of a theoretically errorless administration of the TWE and the two indirect tests (i.e., perfect reliability), the indirect tests could only measure about 37% to 44% of our students' actual writing ability as measured by the TWE. This means that, given a student with truly average writing ability, the use of these indirect measures of writing ability does not allow us to decide with confidence whether the student is at the 5th percentile of writing ability compared with peers (very poor), at the 95th percentile (very good), or somewhere in between. The error of measurement (as indicated by the 95% confidence interval), even with perfectly reliable indirect testing, is so great that using indirect tests for writing assessment in the deaf and hard-of-hearing population is somewhat like playing writing- score roulette!
By contrast, with a single administration of a direct measure of writing skills, such as the TWE, the same student would be identified as falling into the middle two quartiles of student abilities (between the 25th and 75th percentiles) with great (95%) confidence. A further advantage to the use of a direct test is that this range of uncertainty can be narrowed considerably by just repeating the direct test one or more times for a given student, whereas no amount of additional testing with the indirect tests would improve the prediction very much for that student.
Interestingly, the field of writing assessment began with the use of direct essay judgments, but indirect tests soon arose as an answer to severe reliability problems associated with direct assessment (Breland, Camp, Jones, Morris, & Rock, 1987). With improved, standardized administration and scoring procedures, these reliability problems have been solved. Given this and our research results, we have now come full circle to recommend that the most accurate and appropriate way to assess the writing competency of deaf and hard-of-hearing students is through the use of a valid and reliable direct essay writing procedure.
References
Notes of Note
Cultural and Language Diversity and the Deaf Experience, edited by
Ila Parasnis of the NTID Department of Applied Language and
Cognition Research, was recently published by Cambridge University
Press (ISBN 0-521-45477-8, $49.95). This book considers in depth
the notion that deaf people are members of a bilingual-bicultural
minority group, whose experiences often overlap with the
experiences of hearing minority group members, but at other times
are unique. Part I presents an overview of mainstream research on
biculturalism and bilingualism. Part II provides conceptual
analyses and research that examine the impact of cultural and
language diversity on the educational and psychosocial experiences
of deaf people. The third and final part offers rich experiential
evidence from deaf community members that brings home the emotional
impact of living in the Deaf and hearing worlds.
Chapters in Part I, Bilingualism-Biculturalism and the Deaf Experience: An Overview, include "On Interpreting the Deaf Experience Within the Context of Cultural and Language Diversity" by Parasnis; "Living With Two Languages and Two Cultures," Francois Grosjean; "Perspectives from the History and the Politics of Bilingualism and Bilingual education in the United States, Kenji Hakuta and Elizabeth Feldman Mostafapour; and "Cognitive and Language Development of Bilingual Children," Josiane Hamers.
Part II, Psychosocial, Cognitive, and Language Experiences of Deaf People, includes "From the Cultural to the Bicultural: The Modern Deaf Community," Carol Padden; "Early Bilingual Llives of Deaf Children," Carol Padden; "Communication Experiences of Deaf People: An Ethnographic Account," Susan Foster; "Marginality, Biculturalism, and Social Identity of Deaf People," R. Greg Emerton; "Attitudes of the Deaf Community Toward Political activity," Gerry Bateman; "Cultural and Language Diversity in the Curriculum: Toward Reflective Practice," Bonnie Meath-Lang; "Minority Empowerment and the Education of Deaf People," Joan Stone; and "Social Assimilation of Deaf High School Students: The Role of School Environment," by Thomas Holcomb.
Part III, The Deaf Experience: Personal Reflections, includes chapters "Growing Up Deaf in Deaf Families: Two Different Experiences," Susan Searls and David Johnston; "Another New Birth: Reflections of a Deaf Native Signer," Patrick Graybill; "Raising Deaf Children in a Hearing Society: Struggles and Challenges for Deaf Signers," Gary Mowl; "In Search of Self: Experiences of a Post-Lingually Deaf African-American," Dianne Brooks; "Living in a Bilingual-Bicultural Family," Lynn Finton; and "On Being Both Hearing and Deaf: My Bilingual-Bicultural Experience," by Patricia Mudgett-DeCaro.
Individuals indicated in bold type above are current members of the college of NTID.
NTID Research Bulletin
The NTID Research Bulletin is published periodically by the Center
for Research, Teaching and Learning, National Technical Institute
for the Deaf, a college of Rochester Institute of Technology. It is
available without charge.
Opinions expressed in the NTID Research Bulletin do not reflect those of NTID or RIT. Your comments, questions, and requests for more information are welcome. See following address.
If you wish a copy of the NTID Papers & Publications 1996 or if you know of colleagues who would enjoy receiving the NTID Research Bulletin, please send names and addresses to:
NTID Research Bulletin, Bldg. 60-2238
52 Lomb Memorial Drive
Rochester, NY 14623-5604
Fax: 585-475-6500, E-mail:
ASKCRTL@RIT.EDU
Marc Marschark, Director, CRTL
Gail Kovalik, Editor
Center for Research, Teaching and Learning
National Technical Institute for the Deaf