Carnegie Mellon University's TechBridgeWorld research group today announced the release of open source software that can help educators of children with hearing disabilities create video dictionaries of sign languages and use games that encourage vocalization by children learning to speak.    

The software, produced in collaboration with the Mathru Educational Trust for the Blind in Bangalore, India, is particularly suited for use in underdeveloped countries where educational resources for children with disabilities are often in short supply.           

Both platforms — Sign Book for sign language dictionaries and Speak Up! for speech-training games — were created this summer by CMU students participating in TechBridgeWorld's innovative Student Technology ExPerience, or iSTEP. This internship gives students an opportunity to conduct technology research abroad that fills unmet needs in the local communities where they are working. Mathru was a partner in this year's iSTEP.

Downloads of the software are available on the iStep downloads page.    

The Sign Book project was launched because teachers at Mathru's School for the Deaf lacked sign language dictionaries for local languages, said Erik Pintar, a fifth-year senior in human-computer interaction and electrical and computer engineering. Many reference materials are available for teaching American Sign Language, he noted, but there are more than 300 documented sign languages, each with its own signs and grammar.           

Pintar, who was born with hearing loss and learned American Sign Language as a preschooler, worked with the Mathru teachers to create a tool allowing them to make a custom video dictionary of the local sign language, with the ability to capture sign videos and relevant pictures for each entry and categorize entries by topic.

The platform was also designed so educators can readily create dictionaries for other sign languages used wherever they may teach.

Speak Up! was developed to help compensate for the lack of speech therapists as teachers train hearing-impaired students in first through fourth grades to vocalize. Much of the teachers' time is spent getting the children to generate sounds, said Amal Nanavati, a sophomore computer science major. The teachers often make loud noises themselves to prompt the children, which can be exhausting, he added.

So Nanavati, along with iSTEP teammates Maya Lassiter, a junior electrical and computer engineering major, and Minnar Xie, a fifth-year senior in humanities and arts and in human-computer interaction, developed a series of games that would motivate students to make sounds and give them visual feedback so they could tell the volume and pitch of their sounds, or if they were making any sound at all.

A typical game might cause a picture to appear on a screen if the child makes a noise. In another, emitting a sound would cause a rickshaw to move. When the rickshaw is climbing a hill, a sustained, loud sound is necessary to keep it moving forward. Other goal-oriented games will cause a bird to fly at different altitudes, depending on the pitch of the sound the student makes, or for a fish to swim up or down depending on whether the volume is high or low.

"Teachers would use these in their classes to keep students more engaged," Nanavati said.

Most similar games that already exist are designed for use by speech therapists and require professional setup, he added. Speak Up! includes 10 games that can be used by teachers with little or no speech training and that teachers can adapt to fit their curriculum or reflect the local culture.

"Sign Book and Speak Up! were developed in such a way to empower teachers and enhance their teaching experience," said M. Bernardine Dias, associate research professor of robotics and TechBridgeWorld founder and director. "By providing teachers with the tools to create their own signs and come up with their own games, these solutions have the potential to be relevant for different communities and their unique needs. By releasing the software open source, our hope is that other educators will find our tools useful and that the open source community will build on the work we started."

"iSTEP combined a lot of my passions with educational Web development, sign language and working with underserved communities," Pintar said. He said the internship has piqued his interest in studying educational technology.

Previous TechBridgeWorld projects now available as open source downloads include the hardware specifications and software for the Braille Writing Tutor, a device to help teach students to write in Braille using a slate and stylus. The Braille Writing Tutor won the 2014 Louis Braille Touch of Genius Prize for Innovation.