In an increasingly connected industry, many processes can be mapped virtually. “Mixed Reality” is interesting for all areas of industrial manufacturing – from plant planning to customer contact. Therefore, Augmented Reality (AR) is not only important for the gaming, entertainment and telecommunications industries but also for Industry 4.0.
Mixed Reality, Virtual Reality & Augmented Reality
When it comes to digital tools for Industry 4.0, Augmented and Virtual reality applications are regularly among the possibilities mentioned. However, compared to many other technologies, their impact is considvaered to be low.
A look at the results of the 2020 Deloitte Insights Readiness Report shows which technologies are of greater importance to companies:
- The Internet of Things (IoT), artificial intelligence (AI), cloud infrastructures and big data or big data analytics are internationally considered the most important pillars for Industry 4.0.
- This is followed by nanotechnologies, advanced robotics and sensors.
- According to the report, Augmented Reality is among the less relevant technologies, along with 3D printing, quantum and edge computing.
However, the use of virtual tools offers numerous possibilities for the entire industrial production.
Augmented Reality – what is it?
Ever since the success of the game app Pokémon Go, Augmented Reality has become a household word for most people. AR is a popular tool to connect the real world and digital information and is often used in computer and video games, photo and video filters of social media applications or in education and learning projects
This is the essential task that augmented reality fulfils: With its help, real environments can be supplemented or overlaid with digital elements (visual or sound). It requires only the necessary equipment. Because the applications can be used with special AR glasses as well as with mobile devices such as smartphones and tablets, it is very user-friendly.
The basic requirements are:
- the combination of virtual and real objects within a real environment,
- the interactive operation and
- the real-time registration of virtual and real objects with each other.
Different AR methods for visualisation
Markers are usually used for the functioning of AR systems. These help to correctly assess the position and orientation of the virtual objects in the real environment. Alternatively, there are methods that do not require markers, such as Natural Feature Tracking (NFT) or Simultaneous Localisation and Mapping (SLAM).
These various systems are used for the visualisation:
- Head Mounted Displays (HMD) are the famous glasses or helmets. They are typically used for VR applications with completely virtual environments – but can also be used for augmented reality.
- Mobile displays, i.e. smartphones and tablets, are also known outside the industrial context, and some applicationpdfs have already been mentioned at the beginning. In production, mobile devices help to map complex products or processes in a simple way.
- Spatial Augmented Reality (SAR) systems work differently because they use projectors. They project virtual elements directly onto the real objects.
Spatial computing in the context of AR applications means superimposing different virtual layers. The required data can be stored, for example, in an AR cloud. Thus, the visual representation can be connected with real-time data from sensors, production key figures or information from other IoT objects.
The advantage is that this form of visualisation enables hands-free solutions, which means that manual work processes can be carried out while being provided with digital information or being guided by remote assistant systems.
Working in Mixed Reality
Augmented Reality is part of the Mixed Reality. It describes the connection of real environments with virtual ones; real as in the factory halls, machines and personnel and virtual as in data from a wide variety of sources.
AR applications show how this connection looks like in practice. Augmented Virtuality is a subcategory of Mixed Reality. While Augmented Virtuality applications and tools have, like Virtual Reality, a stronger focus on the virtual level, they are still connect to the real environment.
They are an important factor in planning because, for example, new peripherals (such as gloves) can help to capture ergonomics and performance metrics for workplace design. Whereas the application fields for Augmented Reality are primarily in manufacturing.
Strengths and weaknesses of AR in production
The scepticism of industrial companies towards AR applications in manufacturing stems from the fact that their influence could not be assessed sufficiently. Augmented reality cannot be implemented without the appropriate technical equipment, so the necessary investments should show positive effects.
Collaborative research project investigates effect of AR
The impact of AR on industrial manufacturing has been investigated by a research team from the Technical University of Munich (TUM), the University of Wisconsin-Madison and the Mainz State Hospital (Seeing the Bigger Picture? Ramping up Production with the Use of Augmented Reality. Manufacturing & Service Operations Management 2022) in joint field trials. The questions of the project, which was carried out by a technology company:
- How quickly do employees learn to perform new tasks with and without AR support?
- Does the complexity of the task play a role?
- How do AR applications influence the ability of employees to make suggestions for optimising processes?
The impact of AR applications on productivity is the most obvious. The researchers were able to observe that even complex tasks were completed significantly faster with the help of AR glasses (time savings of 44 per cent on average). Even simple tasks took around 15 per cent less time with AR support.
Correctly assessing the advantages and disadvantages of AR
The disadvantage: the technical assistance reduces the learning effect. The information needed to complete the task is not absorbed in the same depth as in analogue working. This in turn also has an effect on innovation potential – according to the results of the study, more helpful suggestions for improvement came from the group that worked without AR technology.
According to the researchers, AR applications have the most positive impact in areas where process optimisation is largely or fully completed. AR also supports production areas where a short tact time is required, thus increasing productivity.
However, many companies find very different uses for AR technologies within industrial manufacturing.
Application areas for AR technology in Industry 4.0
Although, AR applications are somewhat behind compared to other key industries of Industry 4.0, many manufacturers are already using the possibilities of Augmented Reality. The areas of application are very diverse and are by no means limited to making well-coordinated production processes run even faster.
Support with complex assembly tasks
As the above study by the research team led by Prof. Dr. David Wuttke has already shown, AR means making complex task considerably easier. This is also the case in the assembly line, as products and processes become increasingly complex.
A multitude of components must be assembled with extreme precision and under time pressure. The challenge remains the same, no matter the size of the finished product.
However, new products make the assembly more difficult, thus delaying the processes. The reasons for this are, amongst other things, new assembly instructions and the way they are presented. PDF documents, which are often used to gain the necessary information, take up too much time.
However, with AR support, the information can be made easily accessible and highly visible. Using AR support with voice control also helps speed up the process as workers have their hands free and do not need to interrupt the assembly to gain the information needed.
This way of using AR is not new, however, technological progress has made the necessary equipment and associated software much cheaper than it was 20 years ago. As a result, such AR applications are now more broadly used. In aerospace technology, for example, where assembly is carried out in considerably larger dimensions, projectors are used to show the exact position of individual parts.
Simplification of maintenance processes
Maintenance of production facilities is a critical factor from an economic perspective. Industry 4.0 aims to reduce unexpected production downtimes by automising production facilities and planning their maintenance.
Even without the possibilities of largely or fully automated machine maintenance, time plays a central role. If maintenance results and targets have to be transferred into manuals, the processes are delayed and take longer than necessary.
With AR technology, the comparison of goals and results can take place in parallel on an AR display. In this case, the voice control also helps to use the manual easily while the hands remain free.
HoloLens technology combines maintenance service, virtual models and modern communication channels. Thus, maintenance technicians have access to all relevant information (such as technical data, etc.), can compare these with virtual models and carry out maintenance with their hands free.
The technical support does not even have to be on site to do this: all necessary inspection steps can be transmitted to an on-site technician via HoloLens. The AR glasses work in two directions:
- The technician on site sees the exact instructions for professional maintenance via the display and receives precise step-by-step instructions.
- The support team at the head office has a direct view via the glasses and can carry out the maintenance remotely.
Together with the data from the models, such interventions can be planned in advance and completed much faster.
Promoting collaborative ways of working
The HoloLens application anticipates another possibility for AR applications: improving collaborative solutions and easier access to expert knowledge. This makes it easier to coordinate and carry out maintenance and servicing work, even in larger production contexts.
AR technology is intended to close the personnel gap that exists in many companies due to the lack of skilled workers. Thanks to AR applications, experts can be called in from anywhere without having to be on site. Therefore, it is no longer necessary to have the required expertise available in house.
The employees can be guided by the experts thanks to telepresence and transmission of the situation in the factory. This form of collaboration can be carried across departments, divisions and plants. This knowledge transfer can even be used for training purposes – as a kind of “learning-by-doing” under the direct supervision of qualified technical staff.
Improving customer relations
Similarily, manufacturers can use AR applications to support customers. Particularly when it comes to advice which machine and plant manufacturers have to provide for complex products.
AR technology enables much better visualisation – of the products themselves, of individual components or of the integration of the machines and systems into the operation. Therefore, customers do not only gain an idea of how a product works, but also of its spatial dimensions at the intended location.
In addition, change requests can be discussed directly with the customer and transferred to the virtual models.
Help with quality assurance
Virtual models of products and components can also be used in quality assurance together with AR devices. The displays can be used to compare the original and the model via overlay. If there are deviations from the manufacturer’s specifications, these can be marked directly in the overlay.
This also applies to larger projects. At Airbus, for example, Mixed Reality applications are used to provide technicians with complete digital models and sketches, in order for them to be able to check structural mounts during production. This helps to significantly shorten the inspection of up to 80,000 brackets.
Better planning of production areas
Using AR technology, models of production areas or entire manufacturing plants can be projected onto the real environment. In this way, it is possible to check the planning of machines, storage areas, transport, travel and escape routes in advance of commissioning – and to optimise them should the need for subsequent improvements become apparent on site.
Application examples for AR: Practical Insights
The areas of application for AR technology described above are no longer theoretical considerations in many companies. In fact, they are already considered an integral part of everyday working life.
Installation of new production lines at Toyota Motor Corporation
Toyota Motor Corporation is using AR applications to simplify the maintenance and the installation of new and existing production lines and factory buildings at its global locations. Technology helps on several levels.
Challenges
- Presence of experts: Checking the work, solving potentially dangerous problems, carrying out elaborate work such as measuring large machines – many of these processes usually require the presence of experts and/or licensed technicians who have to travel to the international sites specifically for this purpose.
- Strict communication guidelines: Strict rules apply to the handling of confidential data in all Toyota operations, from headquarters to local branches. This also includes software, cameras and other end devices that open up possible security gaps. This makes communication between distant locations more difficult.
Solutions
- Secure equipment: To improve communication and the flow of information, Toyota relies on secure tools. These include, for example, Vuforia Chalk (from PTC) for remote assistance.
The tool comes with a range of features such as live video and audio, as well as the ability to add digital remarks to the real-time view. To ensure safety, the tool was customised for Toyota and pre-installed on mobile devices that are available for loan in the companies.
- Remote experts via AR: The use of AR technology at Toyota is a classic example of how it can be used to make expert knowledge for a specific problem available over long distances. All work steps during installation and maintenance can thus be accompanied directly from a distance via mobile phone or tablet. The technician on site is supported with precise instructions.
Not only does it reduce the amount of travelling, but also the time that would otherwise be needed for machine measurements. In addition, errors in implementation can be avoided with this procedure.
Augmented reality for training purposes at the BMW Group
Contrary to the results of the TUM study on meaningful applications of AR, the BMW Group has found an approach to using the technology to train employees. In fact, apart from coordinating picking operations in logistics and quality assurance (such as in the paint shop), it is the most important use for AR applications.
With the help of videos and instructions superimposed on the real environment of the shop floor, the numerous assembly steps are to be conveyed. The videos can be adapted to the respective employee and can be fast-forwarded and rewound as desired.
A trainer accompanies the exercises under live conditions. They can answer specific questions that go beyond the video and instructional content. AR technology makes it possible to train several employees at the same time.
According to the company, the training offer is well received. Overall, the training videos ensure that the desired learning goals are achieved more quickly. The assessment is different for assembly: Here, the BMW Group does not see any added value for AR applications, although there are certainly application possibilities. However, the focus in this area is also on lean processes.
HoloLens pilot project at Škoda
A pilot project was launched at Škoda in 2021 to test the various possible uses for AR glasses. Areas of application include, but are not limited to:
- maintenance to be optimised through the immediate projection of manuals, maintenance checklists and similar documents, while the technicians’ hands remain free;
- Training courses that can be supplemented by parallel video conferences, for example.
The targeted goals of the AR project are also different. They range from higher occupational safety (for example through a lower error rate) to improved communication across departments and locations to shorter maintenance processes.
Gamification elements should also serve as an incentive for younger employees and facilitate access to technical professions.
AR for employee retention at the Volvo Group
Just like Toyota Motor Corporation, Volvo Group relies on PTC’s Vuforia tool for its AR applications. The technology is used in various areas of the company. One important goal is to position Volvo as an attractive, innovative employer.
Augmented Reality plays a major role in this because, according to the company, it can appeal to different groups of employees and applicants:
- Existing employees are actively involved in the new processes and thus contribute to the transfer of knowledge. Especially in the quality control and assurance of engines, this makes it possible to collect the know-how of long-standing experts and to prepare it in a contemporary way for junior staff.
- AR technologies are generally of great interest to applicants. By using appropriate tools, the Volvo Group benefits in two ways.
On the one hand, they strengthen the brand as a modern technology company and thus its attractiveness as an employer. On the other hand, AR applications significantly facilitate and shorten the necessary training of new employees.
All in all, augmented reality is intended to reduce the cognitive load of employees while at the same time promoting the ability to learn. AR technologies are therefore increasingly used in areas that cannot be automated and rely on human skills and analysis.
Versatile toolbox for production
AR technologies are used in many business and production areas. They are used as a support to achieve improvements in quality, costs and time.
To do this, however, it is necessary to adapt the applications to the respective requirements of the different areas. At the same time, it should be noted that Augmented Reality cannot deliver the desired optimisation for every production process.
Nevertheless, the case studies show the strength of AR applications, especially for maintenance, servicing, assembly, quality control and training. In addition, there are overarching benefits such as improving communication within companies as well as with customers and partners worldwide 24/7. In this respect, AR offers a wide range of possible applications that mean added value for industrial production.
