Smart World project image recognition with HoloLens

Their final project was graded with a nine, which is about one of the highest grades in the Dutch grading system. The five minor students of Windesheim University have worked hard on their Smart World project. They were commissioned by Allinq to devise a way to automatically recognize assets in the field using the HoloLens. Roel Hoppenbrouwers; New Technology Consultant and XR developer at Allinq Labs has been supervising the students. Together, we look back on this project.

Presentation to trainees

In September 2019, 52 enthusiastic undergraduate students from Windesheim University of Applied Sciences took part in the exploratory workshop in Allinq Labs. Roel and his colleagues then highlighted exciting themes such as Augmented Reality, Virtual Reality, Digital Twinning and the use of gaming devices. In addition, a LoRa workshop on programming gaming devices and connecting sensors was given. Roel: “We regularly have meetings like this, where we show young talent what we do. We do this to introduce them to our beautiful profession and to present ourselves as a potential internship. Up until now, we have always received a lot of positive reactions on our minor projects. Especially the HoloLens project, which they liked the most, was soon occupied.”

The assignment – automatic recognition

“What options are there to automatically recognize objects (assets) in the field of a Field Engineer using the HoloLens and to link a specific work instruction to the relevant asset?” This was the very broad ‘learn by doing’-assignment that the five selected students received. Outside of the two school days, the team had full access to the Allinq lab for six months. Weekly progress was discussed with Roel. Roel: “We deliberately do not delineate the assignment completely, so that the students can choose an application themselves. They are undergraduate students, so we expect them to work on this project independently. If I’d explain it, there would be an immediate solution. However, I would like to know how they would do it. They learn from that and that is much more interesting for us.” Through the Microsoft Teams collaboration platform, ideas were exchanged and presented in blogs and peer reviews to other Smart World teams and the school management.

The implementation app for HoloLens

By working closely together, the team has achieved wonderful results in a relatively short amount of time. Roel: “They have made an app for the HoloLens. As soon as you take a photo with it, it recognizes which object is in front of the lens and opens a PDF with the information about the relevant asset. The dashboard neatly keeps track of how often something is scanned. This way you collect data about the question. What they have delivered is so modular that we can easily build on it step by step. They did a wonderful job. The interface isn’t that good yet, but that wasn’t the assignment either. This gives us a good basis to build on and we are fully engaged in that.”

The sequel – Augmented Reality Installation Guide

With a beautiful nine as a final grade, the Smart World team has now completed its project. The result does not end up in a dark desk drawer but is now in the hands of Roel and his colleagues. Roel: “We have already done some things with it. For example, we used parts of the code to also integrate image recognition into the Augmented Reality-installation manual of one of our customers. We had discovered that many end users initially indicated that they wanted to do the installation of their modems themselves, but that they eventually ended up asking for a technician. The paper manual turned out to be too general; which meant that the end user had to be able to estimate many aspects himself, such as the type of connection that was in his distribution board. We have therefore created a smart manual. The end user downloads the app. With that app and the normal camera of the mobile phone, the end user can photograph his distribution board. The connection point in the distribution board is recognized via image recognition. Then Augmented Reality shows step by step how to install the modem. In fact, this is the basic code for many more great applications in the future.”

The Future – Generative Adversarial Networks

Allinq Labs is now investigating the possibilities for GAN; Generative Adversarial Network, with which the data models can be trained with only a limited data set. Roel: “A fiber termination unit, the FTU, is a white rectangle with few identifying points. That is difficult to recognize for an app. The data model can be trained with thousands of photos in order to increase the recognition rate. With a GAN, this could theoretically be done with just a few photos. Every time we demonstrate what is possible with a Proof of Concept and try to think further about what else is attainable.” And those options are endless, within the Allinq Group alone. ​​“As for Allinq Insite: They digitize telecom assets such as telecom towers by scanning them and converting them into detailed 3D models. We actually use the same data, so we can help each other effectively. They can help us with digital twinning and we can help them with, for example, virtual reality to visualize their scanned towers.”

Hardware as a limiting factor

Roel: “The HoloLens is really advanced. It is smart, standalone can create a 3D map of the area by scanning the space several times per second using the same technology as the Kinect. It makes everything you put into the world virtual. If you link this technology to the Microsoft Azure Services with image recognition, cognitive services, custom vision and our own data models… that is the technology of the future, I really believe in that. The software can already do everything. However, the hardware is now the limiting factor. We are still working with the HoloLens 1; a technique that is now almost five years old. It is not ideal for use in the field. The HoloLens 2 already forms a better basis. It has been ordered, but is not yet available. AR developments are also fast-paced: there are many frameworks to use conveniently from your phone, but then you don’t have your hands free. Mirroring your phone screen somewhere on your helmet is also a nice opportunity. There are still many developments, but the technology we have in our hands is way ahead of time.

Internship at Allinq Labs?

Are you curious about the possibilities of an internship at Allinq Labs; we are always open to the energy of curious technical students! Ask us if there is an assignment ready or propose an interesting theme to us! Please contact Allinq Labs manager Rutger van der Graaff at r.van.der.graaff@allinq.nl!