Mixed reality glasses will displace smartphones

Mixed reality combines the physical world with digital objects and is used in the business environment for maintenance work and education. Market researchers are predicting a great career for the technology. "In around 10 years, mixed reality glasses will replace standard smartphones," explains ETH Professor Marc Pollefeys. The glasses are then likely to characterise the street scene, and early adopters will be around in just a few years.

Pollefeys heads the Computer Vision and Geometry Lab at ETH Zurich and the Microsoft MR & AI Lab, which opened in 2019. In addition to writing scientific papers, the researchers are also involved in the development of specific products.

An anchor from Zurich for the mixed reality

At the Ignite in-house exhibition in March, Microsoft made it clear that it sees its future in mixed reality. The company presented its "Microsoft Mesh" platform, which allows users to enter a "shared holographic world". This requires so-called spatial anchors, without which users would not see their digital content in the same physical location. The shared mixed reality would be split into a multitude of individual worlds.

The researchers are creating comprehensive maps so that the wearers of MR headsets and the information can be correctly localised. This is a prerequisite so that not only Hololens wearers or smartphone users in MR mode can see the same digital objects in the same place, it should also enable the use of robots. "We carried out a project in which people only had to point to certain places in mixed reality and a robot did its work there," explains Pollefeys with pride. This could become a big hit in factories.

In any case, the lab manager sees a wide range of potential applications for the MR devices: In addition to the areas of maintenance and training, where the technology is already widespread, there is a whole host of use cases in medicine, for example. "Mixed reality can be used particularly well wherever people have to perform complex tasks but there are few experts," says Pollefeys. The specialists would not have to be flown in specially, but could assist remotely and be virtually on site.

Hurdles: Privacy, standards and hardware

Pollefeys describes the smartphone, where you always need your hands, as "indirect augmented reality". The technology will never experience a breakthrough on these devices. The full benefits can only be realised with the new generation of hardware, the mixed reality glasses. But that's just it: It will probably be another decade before these are fully established.

However, Pollefey sees the biggest hurdles on the software side: there is still a lot of work to be done in the area of mapping, for example, the core expertise of the Zurich lab. In the future, data will be fed in through a kind of crowdsourcing. This data will then come from people travelling through the world with their glasses and recording their surroundings. This is particularly important in a rapidly changing environment such as a city.

If mixed reality spreads across the business world and conquers the streets, privacy will play a role: The glasses are not only equipped with cameras facing outwards, but also permanently record the user's eyes. After all, the software needs to know where the user is looking. However, no application has access to this, says Pollefeys, who adds: "But you always have to weigh up what information is allowed to be seen by whom in the superimposed layer of reality."

Source: Inside IT / Image: Large-Scale Localisation Computer Vision and Geometry Lab