Learning the finger alphabet with hand tracking

Facebook subsidiary Oculus activated hand and finger tracking for the Oculus Quest at the beginning of the year. The example of Cédric Girardin, a student at Bern University of Applied Sciences, shows that interesting use cases are possible. He developed a VR application in which you can learn the finger alphabet for sign language.

Anyone who learns languages with an app knows the problem. The programme often doesn't give any feedback as to whether you are pronouncing something correctly. Exactly the same problem occurs with sign language. Is the gesture correct?

Cédric Girardin, a student at Bern University of Applied Sciences, programmed a VR app for Oculus Quest that makes learning the finger alphabet much easier than with the naked eye thanks to hand tracking.

Feedback on hand tracking

The application offers a learning mode and a test mode. In learning mode, a letter of the German alphabet is displayed together with a 3D model of the corresponding hand and finger position. Thanks to the Oculus Quest's hand tracking function, you can imitate the figure shown with your own hand in virtual reality and have the match validated by a machine.

The test mode works in a similar way, except that no 3D model is displayed. The learner must recall the finger sign from memory.

Hand tracking still fails with certain gestures

To validate the hand and finger position, the distances of the fingers, their angles to each other and the orientation of the hand are analysed. The application can analyse up to 23 hand signs of the German alphabet.

The letters R, J and Z are missing. The reason: hand tracking is still overburdened with these characters. For the letter R, you have to cross your fingers and J and Z require finger movements. In addition, the tracking still has problems recognising gestures that point away from the camera.

The VR training app is Girardin's final project and was programmed within two weeks at the Computer Perception & Virtual Reality Lab at Bern University of Applied Sciences. Girardin says on Reddit that he will continue to develop the app if the lab sees a need for it. Such an app would definitely have potential, especially if it could incorporate other gesture-based languages.