See-Through Captions Key Image

See-Through Captions

Real-Time Captioning on Transparent Display for Deaf and Hard-of-Hearing People
The 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '21) 2021

Problem

Real-time captioning is a useful technique for deaf and hard-of-hearing (DHH) people to talk to hearing people. When communicating by real-time captioning using a mobile device, the body language cannot be confirmed because the mobile device must be viewed to see the result. When using an AR device, deaf or hard-of-hearing people can see the voice-recognized text while observing the partner's body language. To realize higher-quality communication and overcome the limitations of mobile and augmented-reality devices, real-time captioning that can be used comfortably while maintaining nonverbal communication and preventing incorrect recognition is required.

Solution

We developed a real-time captioning system that utilizes a transparent display and allows people to check the speech-recognition results while seeing their partner during the conversation. In this system, the captions are presented on both sides of the display to address the problem of incorrect ASR, and the highly transparent display makes it possible to see both the body language and the captions. (Transparent display provided by Japan Display Inc.)

Materials

ASSETS '21 Demos

  1. Kenta Yamamoto, Ippei Suzuki, Akihisa Shitara, Yoichi Ochiai. 2021.
    In The 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '21). ACM. (to appear)

arXiv e-prints

  1. Kenta Yamamoto, Ippei Suzuki, Akihisa Shitara, Yoichi Ochiai. 2021.
    arXiv e-prints.

Introduction Video

Project Members

Kenta Yamamoto
Digital Nature Group,
University of Tsukuba (JP)
Ippei Suzuki
Digital Nature Group,
University of Tsukuba (JP)
Akihisa Shitara
Digital Nature Group,
University of Tsukuba (JP)
Yoichi Ochiai
Digital Nature Group,
University of Tsukuba (JP)
Pixie Dust Technologies, Inc.
Three authors contributed equally to this research.