ETH develops augmented reality app for dentists

Textbooks are often difficult to access due to their information density and can be daunting. The AR Osteoclasts app, on the other hand, makes learning easy. The app, programmed by game developer Jascha Grübel from the Game Technology Center at ETH in collaboration with Bernd Stadlinger, Clinic Director of the Department of Oral Surgery at the Centre for Dental Medicine in Zurich and member of the ETH AI Center, introduces budding dentists to the processes of bone remodelling in a fun way.

When students point the camera of their tablet or smartphone at a specific image in the book Communication of Cells, they land on the bone surface next to a blood vessel. In this 600-fold enlarged microcosm, the players control biological and biochemical processes that would otherwise take place automatically. Each of the six game chapters is introduced by a voice explaining the background to bone resorption and the role they play.

Lead developer Grübel also had to familiarise himself with the topic first: "I watched educational films and summarised all the cells involved, their tasks, actions and interactions in a mind map." Because the processes in the body take place automatically, he had to decide which elements of them would be adopted by humans in the game. "For example, when the player taps on the screen, the osteoclast sends out messenger substances to attract stem cells, which develop into osteoblasts that repair the bone," explains Grübel.

Sometimes the players also receive symbolic help: In chapter three, they control two taps - one for hydrochloric acid and one for enzymes - whereby they can dissolve the bone with the right amount of the two substances. Grübel programmes such game mechanisms on Unity - a platform on which games are developed. "For a game, time must also move in the computer and define the next step in the course of the game," Grübel explains. "Unity already provides this background, as well as, for example, physical laws that should apply in the game." Ultimately, all game mechanics are mathematical calculations, says Grübel.

Fabio Zünd, Managing Director of the GTC, describes finding the balance between play, science and education as a particular challenge: "A game has to be fun and motivating. To achieve this, we have to simplify the biological processes without distorting them." This required an intensive dialogue between the game developers and the dentists, researchers and publishing staff. "Certain game moves reminded the researchers of outdated knowledge models. We therefore had to work together to find out how we could represent the current understanding in a meaningful way," explains Grübel.

Another challenge was to ensure that the app runs on as many devices as possible: "At the GTC, we normally focus on research prototypes - that means they only run on one device, but can do crazy things," says Zünd. In contrast, AR Osteoclasts is a product in the app stores that is intended to reach as many people as possible. The developers therefore had to limit the technology in AR Osteoclasts so that the app also runs on older devices. Digital artist Violaine Fayolle scaled down the detailed 3D model of the bone on which the player is located so that the devices do not overheat when the game is displayed. The 3D model comes from the medical film producer interActive Systems, which was involved in the app development alongside the publisher.

The extent to which students are more motivated or even learn better thanks to the app is currently being analysed in an accompanying study. However, both computer scientists see great educational potential in games. "You can teach a lot with games, but it's difficult to know what you're teaching. We need more results from the learning sciences," says Grübel.