FUTURE APPLICATION OF MULTISENSORY MIXED REALITY IN THE HUMAN CYBER—PHYSICAL SYSTEM

Mixed reality as an emerging technology can improve users’ experience. Using this technology, people can interact between virtual objects and the real world. Mixed reality has enormous potential for enhancing the human cyber—physical system for different manufacturing functions: planning, designing, production, monitoring, quality control, training, and maintenance. This study aims to understand the existing development of mixed reality technology in manufacturing by analysing patent publications from the InnovationQ-Plus database. Evaluations of trends in this developing technology have focused on qualitative literature reviews and insufficiently on patent technology analytics. Patents connected to mixed reality will be mapped to give technology experts a better grasp of present progress and insights for future technological development in the industry. Thus 709 patent publications


INTRODUCTION
Businesses have become more competitive because of global competition, and their technology has developed to ensure their long-term viability. Because of recent advances in operating technologies (OT) and information technology (IT), the industrial sector has entered the age of Industry 4.0, and is known as the cyber-physical system (CPS). This revolution has increased efficiency, allowing increasingly diverse and productive production processes to meet the increasing growth expectations of clients. Artificial intelligence (AI) and the industrial Internet of Things (IIoT) enhance traditional processing infrastructure to handle more complex shop floor activities. However, the process of migrating to CPS in real industry is a huge problem. The Reference Architectural Model of Industry 4.0 (RAMI 4.0) is a solution that presents a step-by-step migration technique to enable connectivity and interoperability in order to identify clearly the point where OT meets IT [1]. Although automation technology has a great degree of manufacturing system stability, autonomy, and intelligence, it is difficult to guarantee that technology would replace any operation in manufacturing processes.
According to a McKinsey report [2], machines could replace fewer than 5% of manual tasks and operations, whereas only 60% of all employees have at least 30% of their activities automated. As a result, a possible approach is to allow people to cooperate with the operational cyber-physical system by assuming the position of 'master', and this concept is presented as the fifth industrial revolution, known as the human cyber-physical system (HCPS) [3]. Figure 1 is a conceptual representation of HCPS in the context of adopting Industry 4.0.  Figure 1 represented the 4.0 stage of the Industrial Revolution (from 1st to 4th). Combining physical and cyber systems resulted in the fourth evolution, the cyber-physical system, which is capable of completing, controlling, and optimising operation. Even if the AI system is robust, the human part is still vital. Therefore, the system advances to the fifth level, producing a human cyber-physical system. Finally, it will fully exploit and combine human and machine intellect synergistically [3].
As a result of the manufacturing support system's advanced technology installation, data is generated, evaluated, and shown as visualised information, warnings, or reports. As a result, feedback can be developed to improve human operations in manufacturing [4]. Immersive technology, which includes virtual reality (VR), augmented reality (AR), and mixed reality, is one of the most effective approaches to enhancing human participation and HCPS in the industrial sector, known as mixed reality (MR). MR is a user interface application that employs visualisation methods. It improves the view of the real world and allows for near-simultaneous communication between virtual and real settings. Amplifying knowledge enables remote participants to annotate user views, and improves collabourative interactions. As a result, in the intelligent manufacturing environment, MR has a considerable capability for HCPS.
This study focuses on MR technology, which is generated by mixing virtual reality (VR) and augmented reality (AR). This study aims to answer two research questions: (1) How can existing multisensory MR, based on the patent literature, be applied to manufacturing operations? (2) What are the predominant human senses, and how can the possible future development of multisensory MR for different manufacturing operations be applied in three industries (automotive, food and beverage, and textiles)? MR is a cuttingedge technology that allows for maximum user involvement in both the real world and the virtual world. Various solutions to support MR environments are being developed and used across a wide range of industries. To comprehend better the evolution of technology and MR applications, the authors did a literature review [3,[5][6][7][8][9][10][11][12][13] from the Scopus database to build an MR ontology. We followed earlier research [3] in identifying manufacturing operations in which MR could enhance or facilitate technology or interaction devices. It demonstrated that MR has already implemented numerous production procedures, from planning to maintenance [5][6][7]. In the context of MR technology, it can be split into input and output from the user's perspective [3,[8][9][10][11][12][13]. Both are crucial in the creation of immersive and intelligent systems to support human behaviour/nature. For the technology, we took seven papers as our reference to identify the critical technology of MR. Each author of this article identified the crucial technology in recent papers on MR and classified it. We did an author triangulation to check and validate the findings, and then drew the MR ontology-based literature review in Figure 2 below.

Technology mining and patent analytics
The advancement of science and technology leads to innovation and the development of existing technologies. As a result there is a rapid evolution of technology and increasing publication of intellectual property such as patents. To understand the current technology, people conduct patent analyses. Patent analysis needs specific techniques to analyse the patent documents, such as patent classification, technology clustering, patent management mapping, patent bibliometrics, and patent quality analysis [14,15].
The collection of patent documents is relevant to a specific industrial domain. Technology clustering provides a mechanism to group similar patent documents. This clustering outcome can help one to understand the research and development focus among firms [15,16]. Meanwhile, a patent quality analysis helps the company and inventor to decide whether to extend new research and customise its development [17]. A patent management map makes visual the existing intellectual property roadmap. Using this roadmap, the company and inventor can plan new research and development policies, and recommend new areas of innovation that have more significant potential [16]. The patent map also helps to visualise the distribution of patents, monitor the trend of technological change, and infer patent portfolios' strategies [18]. Furthermore, to understand a patent's quality, the expert uses patent bibliometrics and patent quality analysis as a tool. Patent quality analysis usually includes citation analysis and backward and forward citation [17,18].
People used different applications and platforms to get patent documents, such as Google Patents, InnovationQ Plus, and Derwent Innovation. Those various applications also have other functionalities and features to help users to extract the patent information, such as assignees, patent claim, inventors, the international patent classification (IPC), the cooperative patent classification (CPC), country origin, and much other information.
Patent analytics can be seen as a tool for comprehensive technology mining, for both supervised and unsupervised learning. The aims of patent analytics are to find and understand state-of-the-art technology trends based on the patent documents. By analysing and studying patent documents, people can get more insight into the existing technology and how far it has been developed. It can also decrease the R&D cost by up to 40%, since the patent analysis can avoid a false assessment of product, technology, and service development [19].

Technology function matrix
A technology function matrix (TFM) is one of the patent maps that visualise two different patent analytics parts, technology, and management [20]. A TFM is also an indicator that is constructed to match the technology (represented by patent documents) and functions (represented by non-patent literature) of the specific key terms [21]. For example, people usually draw the technology development from the particular patent documents in the technology part. Thus the technology part aims to gain more innovative ideas and to understand the specific technology's existing development track [18,20]. On the other hand, the expert uses the management part to draw information related to the assignee, country, application date, publication date, IPC, CPC, citation, and function of a specific technology [20,21].
There are some benefits that people can gain by drawing a TFM. First, understanding the TFM can help R&D personnel, innovators, and engineers to know about state-of-the-art technology developments [20]. Second, it can also help to analyse the development phase and implement the specific technology in specific functions [21]. Finally, the TFM can also help to decide future technology development and the prospect of the future implementation of the existing technology [18][19][20].
To draw a TFM, people need expert judgement, and to construct a TFM, people need to identify, read, analyse, and structure the patent documents into the specific intersection between technology and function [20]. That construction job is labour-intensive and time-consuming, and requires a lot of knowledge of the specific domain [21]. TFM construction also requires enormous volumes of patent documents and non-patent literature. The development of text mining methods and applications helps experts to analyse and construct the TFM efficiently [19,21].

Cyber-physical system, human cyber-physical system, and digital-human senses
In reality, physical and human spaces have always interacted. Eventually, the cyber or digital space was integrated, giving rise to the concept of a cyber-physical system (CPS). The integration's evolution has been divided into three stages [22]. Stage one was the traditional human-physical system (HPS), in which operations were entirely manual, and information sensing, analysis, decision-making, and operation control were primarily the responsibility of humans. There was also a limitation in terms of quality, efficiency, and performance. CPS represents the introduction of digital integration as the second stage. During this phase, the cyber system's activities include information sensing, analysis, operation control, and decision-making. The task quality, efficiency, and process stability have all been enhanced. Humans, like technology, are expected to perform at higher levels of competence. Thus the advancement of technology is synonymous with people upgrading their skills, type of work, and status of responsibility.
Furthermore, people will be surrounded by an increasing number of automated systems. Such systems can be used either to replace or to collaborate with human input. In general, full automation occurs when a procedure no longer requires human participation. Semi-automation is defined as a system that considers human insights in the control loop. HCPS is an example of an application that holds human input accountable throughout the process. According to RAMI 4.0, the manufacturing execution system (MES), which acts as a component manager for digitalised assets on the shop floor, plays a key architectural and functional role in CPS [1]. The majority of these attempts have centered on the technological aspect, ignoring the humancentered perspective. The third stage will acquire the capability of generating knowledge, and people working with these technologies will have the opportunity to engage in more creative aspects of the work. Humans' roles have shifted from passive information receivers to information or knowledge generators [23]. Taking human interaction into account, CPS evolved to HCPS to introduce next-generation intelligent manufacturing. To enhance human skill and interaction in HCPS, multisensory technology is needed [24]. It is possible to develop digital human senses by implementing certain technologies in the surrounding human environment (as seen in Figure 3).

Figure 3: Digital-human sense overview (adapted from [25])
The system is composed of both ambient and wearable sensors with different modalities [25]. Each sensor has a unique ability to collect specific information about the human worker. Various ambient sensors are used to capture workers' activities in the workplace. Ambient sensing can contain a large amount of data without interfering with the worker's movements. Nevertheless, the complex setup and occlusion issues are the main challenges in implementing ambient sensing. Therefore, wearable sensing was applied, including HoloLens, EMG (surface electromyography), ECG (electrocardiogram), EEG (electroencephalogram), IMU (inertial measurement unit) sensors, and a smartwatch. This scenario produced data, and all of the data were synchronised and sent to the local workstation or cloud server via different transmission protocols. With the data stored in the cloud server, it could be easily accessed in real time.

RESEARCH METHODOLOGY
As shown in Figure 4, this research's proposed methodology consisted of three main parts: (1) technological specification, in which we briefly discussed the patent portfolio used mainly in MR technology for the manufacturing process; (2) TFM, which was used to match the technology application and manufacturing process, in which we identified six manufacturing processes: planning, designing, production, monitoring, quality control, training, and maintenance [26,27]; (3) future application of a human cyber-physical system; in this part we mapped the future human cyber-physical system and enhanced the user experience by using digital-human senses (visual, auditory, smell, taste, and haptic). To complete our analysis, we mapped these human senses to different industries. We used three other industries as examples to analyse the textile, food and beverage, and car assembly indiustries. We also provided future application development and direction that might enhance and advance the human cyber-physical system's manufacturing processes in those three different industries. ; and a number of other patent offices. We analysed the patent literature in the ten years from 2010 to 2019 for active and granted patent publications. We limited the CPC to narrow down our search. We used the search strategy (see Table 1) from the InnovationQ-Plus database, thus gaining 709 patent documents to use for further analysis.

Technology function matrix
We continued our analysis by using the ontology-based TFM, an approach that reduces the technology mining workforce and enhances patent analysis accuracy and consistency [21]. TFM consists of two indicators: technology and functions. A TFM is developed to match a specific patent's technologies and functions and put it into a single matrix. Thus a TFM can help people to understand technological developments and identify potential research and products [21]. Therefore, in this TFM analysis, we used our patent portfolio in the technology part and matched it with the six functions in the manufacturing process mentioned in subsection 3a. This TFM analysis will be a foundation for our future application development of the HCPS. Therefore, we discuss it briefly in the next subsection.

Future application of human cyber-physical system
In this part we give a broad idea of how to improve the accessibility of HCPS in different manufacturing processes. MR technology provides human-machine collaboration. Therefore, using this feature, humans can collaborate with machines to advance the manufacturing process. We extended our future research direction by adding digital-human sense stimuli (visual, haptic, smell, taste, and auditory). Human sense stimuli are essential in a human cyber-physical system [25]. Adding human sense stimuli to a human cyberphysical system can enhance and advance the user experience. We also provided the human cyber-physical system (HCPS) and digital multisensory implementation in different industries, such as textiles, food and beverage, and automotive, to other manufacturing processes. . According to the IPC code, most of the patents were in the category G, which is related to physics. There were three dominant IPC codes: G06T (image data processing or generation), G06F (electric digital data processing), and G02B (optical elements, systems, or apparatus).

Technology function matrix result
A technology function matrix was developed to match the existing patent technology with the different manufacturing processes. We took the technology from the domain ontology of MR (see Figure 1) that were discussed most: (1) auditory; (2) organic light-emitting diode (OLED); (3) holographic; (4) haptic; (5) eyetracking; (6) head-mounted devices (HMD); and (7) smell and taste sensory technology. On the other hand, we identified seven essential functions in the manufacturing processes: (1) planning; (2) designing; (3) production; (4) monitoring; (5) quality control (QC); (6) training; and (7) maintenance. To create this TFM, we used prior non-patent literature as our training dataset. We also used patent literature as our testing dataset to match technology and function. Table 2 shows the TFM results and the extra column, 'Human senses', to map the technology to specific human senses.
From   Auditory Auditory  66  87  57  103  6  23  17  Visual  OLED  21  20  51  69  2  12  14  Visual  Holographic  74  67  62  167  25  13  50  Touch  Haptic  157  101  98  189  21  22  Eye-tracking is a low-cost and highly accurate technology, but it received the least attention in the manufacturing process [28,29]. Using eye-tracking technology can understand and track the operation process and trigger devices [28,29]. In addition, eye-tracking can be used in different manufacturing functions. For instance, in quality control, eye-tracking technology can control potential defect of a product, especially for a complex product. In training, eye-tracking technology can help to understand and quantify experienced workers' implicit knowledge and transfer it to new staff members [28].
Holographic display technology allows the creation of a virtual 3D image, and distinguishes it from other forms of 3D display. These specific abilities to project 3D images are suitable for holographic display technology in planning, monitoring, quality control, and maintenance. In addition, there are some current uses of holographic display technology in manufacturing. For example, Hennigar et al. [30] proposed holographic technology to design high efficiency and reliable volume phase holographic optical elements. Holographic technology is suitable for this process, since it needs photosensitive materials.
Smell and taste technology, too, can be used in the manufacturing process, especially in the food and beverage-related industry. Currently this technology needs to capture the flavour and fragrance of specific ingredients. Smell and taste technology is also required to help particular industries, such as cigarette and tea companies. For example, smell and taste technology in manufacturing can be used in the monitoring and quality-control process to taste non-destructive technology for detecting insects in fruits, or a hybrid electronic nose to evaluate pasteurised milk [31].
Haptic technology, also known as kinesthetic communication or 3D touch, can create an experience of contact by applying forces, vibrations, and motions on the user. Haptic technology generates an electric current that drives a response to create a vibration. Haptic technology is helpful in the manufacturing process, such as in planning production of remote human-robot collaboration in a hazardous manufacturing environment [32]. Haptic can also evaluate workforce gestures with visual detection and haptic feedback [33].

The development of HCPS in three different industries
Industrial evolution brings an integration of technologies and human cognitive abilities in the manufacturing process. With this integration, people now want not only to focus on its functionalities but also to enhance the human sensations. Using the five human senses (visual, auditory, smell, taste, and touch), existing technologies can meet people's demand for sensation technologies. It also happens in HCPS technology. However, manufacturing industries need some operations that cannot be replaced by technology and that do not have to rely on humans being able to avoid the high mental workload.
Therefore, there is a need for technology to support human activities as if they could do them naturally. Multisensory HCPS technology should provide human sensations for the collaboration between humans and technology in the manufacturing process. Different processes in manufacturing require the main human senses. To understand better the existing multi-modality HCPS technology in three different industries, we use three sectors to discuss the future development of multisensory HCPS: automotive, food and beverages, and textile. Those three sectors are chosen as the examples because they are varied in respect of the required human sensations: the automotive sector mostly needs haptic and touch technology in the manufacturing process; the food and beverage industry requires smell and taste technology; and the textile sector requires visual technology to understand patterns and colours.

Automotive industries
MR, including virtual reality (VR) and/or augmented reality (AR), has been implemented on a wide scale in the automotive industry. This technology has proven benefits for improving safety, performance, and product design, for better training, for the faster execution of activities, for error reduction, and for user satisfaction [34,35]. The literature review found that MR technology has been implemented to support human involvement in systems, also known as human cyber-physical systems (HCPS). Some MR implementations in the automotive industries have been in the following areas: (1) vehicle design and conception (for example, virtual mockups for manipulating design and ergonomic considerations); (2) factory planning (for example, virtual simulation of the entire production system and detecting discrepancies/collisions, and providing in situ visualisation); (3) vehicle production and assembly (to evaluate tasks such as picking, assembly, and quality control; video-mapping the process of picking shipments in storage logistics can shorten the time needed for preparation); (4) marketing and sales support (for instance, a virtual showroom to improve the customer experience by offering an excellent way for customers to view and test drive a car from a distance; it also improves the after-sales experience for buyers who can do self-assisted maintenance work with AR guidance tools and so understand their cars better); (5) maintenance support and inspection tasks, such as remote assistance, collaboration, and automobile servicing to support vehicle diagnostics and repair, thus reducing the learning time and the mental effort; (6) training programmes related to the production process in which participants are helped to work without making unnecessary errors; and improving teams' satisfaction rates while reducing risks and resource loss.
The visual sense is still dominant in automotive applications. However, the industriy has already considered multisensory systems to create a comprehensive experience [36]. Multisensory stimuli closely mimic human perceptions of the environment using the five senses, thus improving the overall awareness of the human operator at work. Furthermore, MR provides flexibility in overcoming the limitations of the manufacturing environment and increases users' comfort level by increasing the amount of information delivered and by reducing the recipient's cognitive load [3,37]. To elaborate on multisensory HCPS in the automotive industry, we illustrate it using the predominant human senses. Most MR technology developments and applications in the automotive industry are a combination of visual, auditory, and haptic; the senses of smell and taste do not yet exist in MR technology. The integration of the senses of visual and smell would possibly support the maintenance process. For example, a combination of the visual and smell senses (odour) in an electronic nose would provide more information so that users could quickly react to a machine's condition, diagnose it, and make a decision [38]. The development of autonomous vehicles is still a hot issue in the automotive industry. Multisensory HCPS could support vehicle testing programmes to provide users with greater safety. Multisensory stimuli (combining the visual, auditory, haptic, and smell senses) could support the comprehensive development and testing of autonomous vehicles: it could superimpose on the driver's field of view the trajectory to support a manoeuvre, suggest directions when navigation decisions are being made, and reduce the cognitive effort of the driver. However, some MR technology implementations have encountered obstacles [35] such as these: (1) registration technology encounters inaccurate tracking systems on a large scale; (2) display devices have problems with reliability and their readiness for hands-free transport; (3) the driver's information display should not confuse or block their view; and (4) the ethical considerations for technology should focus on privacy, security, and risks to accessibility.

Food and beverage industries
The food and beverage industries produce and process different foods, beverages, and dietary supplements [39]. Thus they are among the industries that are multidimensional, complex, and challenging. In addition, these industries have unique characteristics because they manufacture raw materials, semi-finished products, and finished products. Therefore, the future development of human cyber-physical systems might help with the manufacturing functions. Furthermore, the possible future application of HCPS in food and beverage industries might be to design and create artificial flavours that used the human senses, particularly taste, smell, and visual. In this case, a human cyber-physical system could be improved to predict and deliver better the flavours preferred by customers.
Furthermore, since the flavour perception is affected by changing the colour of a food or drink [40,41], it might be possible to provide visual perception through colour in designing an artificial taste in the production process. Another potential application of multisensory HCPS is using auditory and haptic technology to measure and improve a food's texture and sound. The sound perception of a food and beverage product carries some meaning for customers. For example, the crispness of food can give the sense of freshness in many fruits and vegetables, and crunchiness in some potato chips [42]. Multisensory HCPS also helps the food and beverage industry to design packaging as the critical element in performing multiple functions to optimise the customer experience [43]. Different sensory stimuli, such as haptic, auditory technology, or vision through an OLED screen or HMD, can help to design better food packaging by focusing on creating a better customer perception and experience of the food and beverage industry. To elaborate on multisensory HCPS in the food and beverage industries, we draw on the predominant human senses.  Table 4 shows the predominant human senses for multisensory HCPS that could enhance the manufacturing process. We suggest that smell and taste technology could be used in food and beverage designing, production, and quality control. Currently, some smell and taste technologies might possibly be integrated with HCPS -for instance, the electronic nose and the electronic tongue [44], which could be used to design and measure the quality of food and beverage products. They might be complemented by using haptic technology to understand the food's texture, vision to enhance colour perceptions, and auditory technology to develop the sound perceptions of food and beverages. Therefore, multisensory HCPS plays a significant role in design and quality control. In addition, visual technology could be seen as a sense to be used in food and beverage production. Through visuals, it might be significantly easier to convey complex information about the manufacturing process.

Textile industries
The introduction of new fibres and textiles and of cutting-edge processing technologies has fuelled the expansion of the textile industry for several decades. These trends stimulated the industry to be more innovative and creative by adopting advanced technologies. The market has also benefitted from changes in customer lifestyles. Thus the textile and fashion industries have recently integrated human cyber and physical systems to be more user-friendly and to respond to the future challenge of lifestyle needs and requirements, such as being environmentally friendly. This could provide many other advantages, especially in the production process and the customization of textile product.
One of the MR technology applications in the textile industry is virtual prototyping (VP) in 3D dimensions. This application is developed to plan and design the manufacturing process. It can also help to visualise the prospective customer for greater clarity [45]. Production scheduling and monitoring are also implemented in textile industries [46]. In marketing communications, the magic mirrors technology, introduced in 2012, enabled the customer to choose colour options by using a touchscreen while 'trying on' clothes in front of the mirror, customising different garments for the body type, fitting shoes by just pointing the camera at their feet, and looking around a virtual showroom. These technologies generally provided visual perception that was more personalised for the customer.  Table 5 shows the predominant human sense -the visual sense -in the textile industry that is used to improve production and marketing communications. MR technology can strengthen textile industries' robustness and capabilities, and transcend conventional manufacturing technologies in the future. Furthermore, the extended development of the textile industry has made significant progress, especially in relation to smart materials being integrated with micro-electronic systems for wearable applications, better known as STIMES. The study of Shi et al. [47] covered several aspects of STIMES, such as the functional aspects of materials, the manufacturing process for innovative textiles components, available devices, system architectures, heterogeneous integration, wearable applications in human-related and nonhuman categories (haptic technology), and the safety and security of STIMES. Moody et al. [48] explored the potential use of the sense of touch in physical designs and virtual experiences in fashion and in the textiles of the future. Finally, Ramaiah [49] presented a theoretical study of applying innovative materials and IoT applications in textile technology.

CONCLUSION
The development of immersive technology such as MR can help to enhance human collaboration and involvement in cyber-physical systems, especially human cyber-physical systems. As it gained much attention in the current research literature, we used patent publication data from the Innovation-Q Plus database to understand recent technological developments. We found that the United States, China, South Korea, and Japan have dominated the development of MR technology, focusing on image data processing or generation, electric data processing, and optical elements, systems, and apparatus. Furthermore, to understand the mapping between technology and the manufacturing function, we employed an ontologybased TFM to enhance our analysis. As a result of our TFM, we understand that haptic and eye-tracking are the sensory technologies that are most often adopted in MR to use HCPS. However, smell and taste sensory technology, like the most recent technological development, is the least used in MR in manufacturing. Therefore, future growth could be maximised by implementing this technology.
Our analysis of this patent technology text mining continued by analysing the implementation of multisensory human cyber-physical systems. Multisensory HCPS technology can bring human-technology collaboration closer to reality by providing human sensations. We analysed the possible future performance of multisensory HCPS technology in three industries: automotive, food and beverage, and textile. As a result, we identified possible future multisensory MR in those three industries, and suggested that it could enhance HCPS in the various stages of the manufacturing functions.
Our study has some limitations. First, the dataset used for this analysis came only from the patent literature. We specifically analysed publications on patents because that could capture the recent developments in the technology. However, we consider that the non-patent literature is also important to understand the current research, mainly from universities and research institutes. Therefore, we suggest conducting future research by adding the non-patent literature to the dataset. Second, our methodology on the TFM could be enhanced to give a deeper understanding of MR development in manufacturing. Therefore, we propose that future research combine the TFM with other text-mining methodologies. Third, the predominant human senses for HCPS that we identified in three different industries could be regarded as our conceptual contribution to the further development of HCPS. Therefore, future research could develop HCPS applications in various sectors and in other functions by referring to our work.