Understanding the deaf with AR

Artificial intelligence and augmented reality (AR) in combination could one day facilitate communication between hearing and deaf people. A student has developed an app that recognises and translates finger alphabet gestures in real time.

Artificial intelligence programmes already help the deaf and deaf-mute to communicate by translating spoken language into written language. The reverse communication path, i.e. from the deaf to the hearing person, is more difficult because the majority understand neither the finger alphabet nor sign language.

Ricardo Santana is studying aerospace engineering at Delft University of Technology (Netherlands), where he is working on AI technology, among other things. In one of his research projects, Santana trained an AI to recognise characters from the North American finger alphabet.

AR glasses help to understand

With the application, he can display any letter using gestures alone. The AI recognises the characters in real time and displays the letter in question above the hand.

If AR glasses become established one day, they could translate gestures directly in one's own field of vision and in this way help with communication. This has a big impact on many people: According to Human Rights Watch, more than 70 million people worldwide use sign languages. For them, AR would remove a major barrier.

Quest hand tracking allows deaf communication

The technology should also help hearing people to learn the finger alphabet or sign language. We have already seen the first innovations in this direction with virtual reality: A Swiss student has developed a VR app for the Quest in which you can practise the finger alphabet by copying the signs with your own hands, as we reported earlier.

The fact that VR and AR can support people with various physical limitations is shown in various applications. For example, there are also various approaches for blind people in which VR is supposed to enable "seeing" or "exploring spaces", such as various VR applications. Reports have already shown.

Source: Mixed