More perspective thanks to augmented reality

Operations on the eye are precision work. With a new simulation platform based on augmented reality, doctors can practise operations three-dimensionally on a virtual model.

For months, budding ophthalmologists have been visiting the lab of Marino Menozzi, associate professor in the Department of Consumer Behaviour, to practise their hand skills for epiretinal gliosis surgery. Such surgery becomes necessary when the connective tissue membrane around the vitreous body of the eye becomes cloudy and exerts traction on the retina, which can lead to tears. The result: severely impaired vision.

Difficult operation with AR

Such an operation needs a lot of tact and experience. Therefore, future doctors often assist for years and additionally practise the operation on plastic models and on live animals. The latter is time-consuming and raises ethical questions; moreover, difficult movements cannot be practised repeatedly.

This is where the Robocop goggles come into play in Menozzi's lab: the training is virtual and with the help of augmented reality. If you put on the glasses, a virtual eye appears floating in the air, magnified eight times - exactly the magnification with which surgeons normally operate under the microscope. The magnified eye is displayed as well as instructions for the operation that appear in the field of vision. For example, arrows, lines and circles that show the trainees an ideal path to follow with the forceps. The real environment appears around the magnified eye, which is why the test persons can also see their hands and the position of the surgical instrument.

Since every hand movement is recorded by the mini cameras on the board and forwarded to the glasses, the computer can calculate exactly how accurately a subject has worked. In addition, budding surgeons can undergo virtual "intelligent tutoring". In other words, an algorithm calculates which stages of the operation were performed successfully and which were difficult. Based on this, a training sequence is put together that promises the maximum learning effect. "We hope that we can improve the training effect so that future doctors are ready for operations more quickly and make fewer mistakes," says Menozzi.

Lack of haptics and delay

Currently, the benefits and acceptance of the simulation are being tested within the framework of three master's theses and with 23 trainee doctors. Gian-Luca Köchli is one of them. He is in the final year of his medical studies and has already tested the AR simulator six times. His conclusion: "The simulation is very sensitive and accurate; I was amazed at how realistic it already seems." However, he found it disturbing that the images on the glasses appear slightly delayed in relation to the actual movement. In addition, a realistic haptic is still missing. In principle, however, he finds the approach of practising operations on the simulator promising before using it on humans.

Menozzi currently sees three major challenges: Since the virtual images, depending on the hand movements, are first calculated by a computer and sent to the glasses via WLAN, they appear, as mentioned, 20 to 30 milliseconds delayed. This means that the latency is still too high, Menozzi notes. Second challenge: Some test persons react to certain simulations with dizziness or even vomiting. The third challenge is the feeling of presence during the simulation. "It is an important indication for us of how well the results can be transferred to reality during a simulation," says Menozzi. At the moment, for example, the field of vision is still very limited. While the natural field of vision covers an angle of about 200°, with the mixed reality glasses it is only 35°. For a realistic impression, 120° would be necessary. Acoustics, smell and haptics could also enhance the presence.

"App Store" for simulations

Sandro Ropelato, who is developing the simulator as part of his doctoral thesis, is convinced that the technology's potential extends far beyond eye surgery. In future, other tricky and high-risk operations could also be modelled in advance for the AR simulation, so that doctors could practise all the steps in the virtual model. He sees further areas of application in electronics, for example in the preparation of silicon wafers for microchips.

Ropelato's longer-term vision: a kind of "app store" with different simulations for his hardware setup, consisting of Hololens, microcameras and computers. The 3D simulation is based on "Unity", which is actually a gaming software, but which has become popular for all kinds of AR and VR applications. Therefore, thousands of software elements are available on the web, which only have to be adapted for one's own applications. Ropelato and Menozzi are planning a spin-off in the medium term to market the simulation platform. "Before that, however, we still have a lot to learn from the users," Menozzi emphasises. "Their acceptance is crucial for our success."

Source: ethz