CaptionMask

The CaptionMask project took shape when there was a lack of a publicly available, safe, and adequately functional solution that helped people who are Deaf and hard of hearing to access day-to-day spoken conversations in a masked COVID-19 world. Most people wear surgical or KN-95 masks, which are a barrier to lipreading. They also muffle the wearer's voice, causing speech comprehension issues for deaf people. While clear masks help facilitate lipreading, not many people wear them. Real-time speech-to-text transcription apps on mobile phones require deaf people to split their attention between the speaker and the mobile phone screens, which forces them to miss non-verbal communication cues such as facial expressions, body language, and other communication signals. CaptionMask is a wearable prototype to facilitate easy access in everyday conversations that uses Automatic Speech Recognition (ASR), a Raspberry Pi, and a 3D printed case/cloth to deliver real-time captions on a face mask. As the world looks to ease out of the pandemic, we believe that CaptionMask would still be valuable in several use cases where people who use spoken communication have to interact frequently with deaf people while wearing face masks, such as in healthcare environments. Moreover, CaptionMask can also be helpful for both deaf and hearing individuals who need to wear facemasks in work contexts, such as in industrial clean rooms, laboratories, chemical plants, food processing centers, and healthcare facilities. Deaf people, as well as hearing people, would benefit from this CaptionMask in situations where they cannot lipread or when speech audio gets drowned in background noise.