1 Introduction

In recent years, the industry has undergone a transformation through Industry 4.0, characterized by a heightened level of automation. This automation, coupled with a human-centric strategy, has been implemented to address social, environmental, and technical challenges, thereby providing more agile and resilient manufacturing systems. It represents the current phase of the industrial revolution, and it is characterized by the integration of emerging technologies such as artificial intelligence, internet of things, and robotics, to facilitate more efficient and effective manufacturing processes by allowing humans to get better decisions. In this context, the concept of digital twin (DT) has emerged as a promising technology that can bridge the gap between the physical and digital worlds. Digital twin is a virtual replica of a physical object or process—also called physical twin (PT)—which can be used to simulate and analyze its behavior under different conditions [1]. The concept of digital twin, first appeared in the early 2000 s. It has become very important with the rise of the 4th industrial revolution, and it is now a pillar of the 5th industrial revolution [2, 3].

Digital twin concept and associated technologies have garnered a significant interest and have seen a multitude of improvements, updates, and contributions. Its applications have extended to various fields, including healthcare [4], construction [5], smart cities [6], agriculture [7], supply chain [8], and manufacturing [9].

At the same time, the 5th industrial revolution has known a significant increase in the use of robotics. Initially designed as simple automatons performing repetitive tasks, robots are becoming real assistants for operators [10]. Their perception, analysis and action capabilities enable them to interact intelligently with their environment. “Cobot” is now used to emphasize the symbiosis between machine and human: the robot collects information, shares it with operators and other intelligent machines, and learns from the human with the main objectives of improving their condition of work and efficiency.

Recently, extended reality (XR) technologies have been integrated with digital twins and robotics, enabling users to interact and manipulate virtual objects and to control robots in real time [11, 12]. This integration has opened up new possibilities for designing, testing, and optimizing industrial systems [13], and has contributed to the development of more advanced human-robot collaborations [14].

Despite the numerous contributions in the fields of digital twin, robotics, and extended reality, to the best of our knowledge only one single publication in 2019 intends to tackle this issue of reviewing the benefits of interfacing extended reality technologies with digital twin and robotics [15]. The authors limit the framework of their study to augmented reality without giving a broad spectrum to the emerging concepts in robotics and extended reality technologies, including augmented reality, virtual reality, and mixed reality.

According to this assessment, this paper, based on a systematic literature review methodology (SLR) aims to provide to the scientific community within the field of digital twin and robotics interfaced with extended reality technologies a comprehensive and exhaustive review of the research works carried out since the last recent years. Key trends, challenges and opportunities will also be identified for strengthening future researches in these fields. Here below are the four main contributions of this paper:

  • Analyze the favored extended reality interfaces for digital twin and robotics interactions, and examine their impacts on application domains and usages.

  • Analyze the digital twin\(\leftrightarrow \)physical twin interactions and their impacts on the usages.

  • Evaluate the extended reality interfaced with digital twin level of applications and pillars in an industrial context.

  • Highlight the main current limitations in physical - digital interactions by introducing the concept of augmented perception, which aims to simulate scenarios where real robots interact with pure virtual objects.

The paper is organized as follows: Sect. 2 outlines the methodology of this systematic literature review. Section 3 presents an overview of the query results and justifies the exclusion of several articles through quantitative analysis. Section 4 presents relevant information extracted from the selected articles and provides answers to these SLR research questions. Section 5 defines the concept of augmented perception and describes its characteristics. Finally, in Sect. 6, the paper concludes by proposing challenges for future researches in the fields of digital twin, robotics, and extended reality technologies.

2 Methodology

The research methodology carried out to conduct this systematic literature review is based on Kitchenham et al. [16] and is illustrated in Fig. 1. The steps involve:

  • Database selection for executing queries (cf. Table 1).

  • Duplicate cleaning for article removal.

  • Application of inclusion and exclusion criteria for selecting relevant papers.

  • Paper full-reading.

  • Quality assessment to qualify papers.

Fig. 1
figure 1

Our SLR methodology inspired by Kitchenham et al. [16]

Full size image
Table 1 Keywords for queries
Full size table

2.1 Planning the review

The objective of this systematic literature review is to explore the applications of Digital Twin and robotics interfaced with extended reality in the context of industrial manufacturing. The subsections below give details about the research questions, literature databases, queries, and the inclusion/exclusion criteria that have been used to filter and qualify the relevant papers. Through the following subsections, knowledge gaps are identified, areas requiring further researches are highlighted, and insights are provided to practitioners and researchers in these fields.

2.1.1 Research questions

To conduct this systematic literature review, specific research questions have been identified, covering the following items: application domains and use cases; favored interfaces; types of interactions; level of applications of extended reality; software; tools; associated challenges.

  • RQ1: What are the manufacturing application domains and use cases integrating digital twin, robotics, and extended reality?

  • RQ2: Which extended reality interface is more suitable for interacting with digital twin and tobotics? What are the impacts of the application domains and usages on the design of these extended reality interfaces?

  • RQ3: What are the main-used digital twin concepts and variants for digital twin\(\leftrightarrow \)Physical Twin interactions? And what are they used for?

  • RQ4: What are the manufacturing application levels and modeling pillars for digital twin and robotic construction interfaced with extended reality?

  • RQ5: What are the software involved in digital twin and robotics modeling, interfaced with extended reality? What communication protocols are involved in digital twin\(\leftrightarrow \)physical twin interactions?

  • RQ6: What are the current trends and challenges in these fields?

2.1.2 Database selection

Search queries were executed on Scopus and Google Scholar by the end of December 2022 on a time-period from 2010 to 2022. Scopus and Google Scholar are both academic literature databases known for their reliability and broad range of scientific sources including journal papers, conference papers, theses, magazine articles, and preprints. 4 meta keyword clusters have been used to carry out this systematic literature review, combining: “digital twin,” “extended reality,” “robotic,” and “manufacturing.”

Despite the large period (12 years) for carrying out this review, the first works, compliant with the research queries, only appeared in 2018. This explains why in the various figures proposed in this article, the dates do not start at 2010 but at the publication of the first works, i.e., 2018.

A detailed analysis of the query results can be found in Sect. 3. The meta query used to extract articles is defined as a combination of logic operators such as (digital twin) AND (eXtended Reality) AND (robotic or manufacturing).

Each of the meta keywords has some variants to cover exhaustively the publications in these fields, as detailed in Table 1.

2.1.3 Application of inclusion and exclusion criteria

First, our literature review has been limited to articles which include the pre-defined keywords in their title. Then, based on the following inclusion and exclusion criteria, articles have been filtered.

Inclusion criteria deal with the ability of the paper to:

  • Tackle digital twin interfaced with extended reality for manufacturing applications.

  • Detail the frameworks, approaches, models and the method.

Exclusions criteria deal with the ability of the paper to:

  • Not tackle digital twin interfaced extended reality for manufacturing applications.

  • All preprints, book chapters (which are not from conference proceedings), editorials, conference abstracts, tutorials, magazine papers.

2.2 Conducting the review

2.2.1 Full reading

After the removal of duplicate results, 57 relevant articles have been identified. Out of these, 32 articles have been selected according to inclusion and exclusion criteria for full-reading and further analysis according to quality assessment questions.

Table 2 Relevant extracted features from the 30 selected articles. j: journal, c: conference
Full size table

2.2.2 Quality assessment

To assess the relevance and comprehensiveness of the selected papers, a quality assessment (QA) procedure [17] is conducted according to several questions, which are described as follows:

  1. 1.

    Are the objectives of the research clearly stated?

  2. 2.

    Is the study designed to achieve these objectives?

  3. 3.

    Is the overall research methodology clearly described in the research?

  4. 4.

    Are the results of the conducted experiments clearly identified and reported?

  5. 5.

    Are the limitations of the current study adequately addressed?

  6. 6.

    Are new perspectives mentioned?

2.3 Feature extraction

Contents of the 32 selected papers have been analyzed to extract specific and relevant information according to 23 features which relate to:

  • Use case: this category refers to the use case. A first sub-category provides a brief overview to give insights about the technological implementation context. Then, use-case activity is reported according to: design, monitoring, programming, training, or prediction. The use of robotics is also tackled and if robotics is used, the human-robot collaboration type and its task (collaboration, assistance, and co-operation) is detailed.

  • Physical twin: this category aims to gather information about the physical twin and the components involved: robots, perception capacities, and other equipments. The first subcategories give insights about the robots: robot suppliers; robotic arm - mobile robot - drone; degree of freedoms. The next sub-categories tackle the perception capacities of the digital twin to collect data: type of sensors involved and whether the instrumentation is embedded or not. The last subcategory gives details about any additional equipments involved in the use case.

  • Communication: this category aims to collect information about the communication direction between digital twin and physical twin and the protocols involved.

  • Digital twin: this category aims to collect information about the digital twin [18]. The first subcategory clarifies the digital twin concept [19, 20], since “digital twin” is a catch-all term with a plethora of definitions and interpretations. Then, various digital twin software are detailed in a next subcategory with clarifications about the application levels. Finally, digital twin modeling pillars are indicated ranging from geometry, physics, behavior, and rules.

  • Extended reality interfaces: this category aims to collect information about extended reality interfaces within the use case by outlining their type (virtual reality, mixed reality, augmented reality) and which digital twin \(\leftrightarrow \) physical twin interactions they allow.

  • Miscellaneous: the two last subcategories of the table refers to additional information that do not match with any of the subcategories, such as challenges and key issues.

Table 2 highlights the paper category, its date of publication, the digital twin concept,Footnote 1 the interfacing of extended reality, and the use of robotics.

After applying the quality assessment procedure on the extracted features, 2 additional papers have been removed, leading to 30 articles finally selected for our systematic literature review.

3 Query result analysis

3.1 Categories of publications

Figure 2 displays a representation of the 57 articles along with the number of occurrences of each paper categories.

First, we notice that the number of papers have strongly increased since the last 5 years from 5 papers in 2018 to 23 in 2022 (+460%) which highlights the growing interest of the community for topics related to extended reality, digital twin, robotics in the field of industry, and manufacturing systems.

Then, the majority of the scientific publications are found to be journal articles and conference papers, indicating a strong emphasis on peer-review and recognition in the scientific community.

In addition, the wide range of publications (book sections, thesis, magazine articles, and preprints) suggests that there are multiple outlets for scientific communications. Each serves a different purpose and has its own set of requirements and audience, which help researchers to reach a larger range of readers for getting a greater impact.

Fig. 2
figure 2

Types of publications from 2018 to 2022

Full size image
Fig. 3
figure 3

Keyword occurrences and their links

Full size image

In detail, in 2018, 5 conference papers have been published. In 2019, 3 conference papers and 3 journal articles have been published, while in 2020, there have been 5 conference papers, 3 journal articles, and 1 magazine article published. The trend continues in 2021, with 1 book section and thesis, 2 magazine articles, 3 conference papers, and 3 journal articles being published. In 2022, there is a noticeable increase in scientific production, with 4 preprints, 3 theses, 6 conference papers, and 10 journal articles being published. The increase in the number of journal articles and conference papers in 2022, in particular, indicates that the subject of digital twin and robotics interfaced with extended reality in manufacturing systems is gaining significant interest.

According to this paper distribution, we focused our review on conference and journal articles, since they represent 78% of all publications from 2018.

Fig. 4
figure 4

Use-case activities and human-robot levels of collaboration

Full size image

3.2 Keyword occurrences

Figure 3 is a representation of keywords realized with VOSviewer [46] which have been extracted from the papers and their links between each of them. As expected due to the topics of the SLR, digital twin as the higher occurrence.

Additionally, the designations “Industry 4.0,” “augmented reality,” and “virtual reality” are also significant keywords, indicating that these technologies have been studied extensively. We can note that the number of occurrences of the keywords AR and VR is similar, unlike the term “mixed reality” which does not appear. Other relevant keywords such as “big data,” “visualization,” “AR authoring,” and “deep learning” also appear to be important research topics. Furthermore, keywords such as “manufacturing,” “simulation,” “human-in-the-loop control,” “collaborative application,” “additive manufacturing,” “CPS,” “ergonomics,” “mobile robot,” “discrete event simulation,” “human-robot interaction,” and “intelligent workshop” suggest that the research covers various application domains and use cases where digital twin is implemented. The connections among digital twin, augmented reality, virtual reality, and and all other relevant keywords support the fact that these technologies are widely used in several contexts. We can identify a transformation of the uses of the XR interfaced DT from industrial system simulation in 2018 to human-centered study (“human in the loop control,” “ergonomics,” “collaborative application”) in 2022. Overall, these findings highlight the growing importance of digital twin and robotics interfaced with extended reality in various industries and applications.

Fig. 5
figure 5

Extended reality technologies and their relation to the usages

Full size image

4 Discussion

In this part, 30 articles are discussed, and research questions are answered according to the categories detailed in Section 2.3 and fully described in a detailed table.

4.1 Manufacturing application domains and use cases integrating digital twin, robotics, and extended reality

This section aims to explore the various domains and use cases within the industry that are currently being addressed using digital twin technology, robotics interfaced with extended reality (RQ1). The domains of interest include layout design [42], workstation design [39], robot programming [9], robot control [25], monitoring [21], prediction [32] and training [43]. Understanding the specific areas within the industry where digital twin technology is being used can provide insights into the potential benefits and limitations of the technology, as well as potential areas for future researches and developments. Additionally, identifying the most common use cases for digital twin technology in these domains can help organizations to determine how they can best leverage the technology to improve their operations and processes. In Fig. 4, 36 use cases use robots in their studies or researches, while 11 of them are not focusing specifically on the use of robots.Footnote 2

The human-robot interactions are identified in the selected articles by using the 6 levels of collaboration introduced by Mukherjee et al. [47]: L0 (Fully Programmed), L1 (Co-Existence), L2 (Assistance), L3 (Cooperation), L4 (Collaboration), L5 (Fully Autonomous).

A stacked bar chart is used to analyze the different levels of human-robot interactions according to the use-case activities. Figure 4 shows the occurrence of each type of interactions: 12 are fully programmed, 9 are fully autonomous, 5 use cases involve collaboration and cooperation, 4 use cases are related to co-existence, 1 is dedicated to assistance, and for 1 use case, no robot is involved.

As examples, Burghardt et al. [9] propose a fully programmed solution. Complex movements carried out by an operator in a virtual environment are first recorded by measurement units, and then these motions are replicated onto the robot. Aschenbrenner et al. [33] propose a fully autonomous robot system based on an AR/VR architecture for data display and monitoring. In Perez et al. [13] a cooperation human-robot solution is performed for a pick-and-place use case. In Choi et al. [26], authors propose a collaboration human-robot solution for a product development. Havard et al. [39] perform a co-existence human-robot solution where robot and human are located in a same place without any interaction. Also, in this case, authors propose an assistance solution since the robot helps the operator to develop the product. Finally, a welding process simulation is proposed by Stravropoulos et al. which do not involve any robot [31].

The analysis of use-case activities reveals that “monitoring” is the most popular category, with 7 out of 10 use cases not involving robots. “Workshop design” represents 8 use cases, mostly involving cooperation between operators and robots. “Robot programming” and “robot control” are the next most commonly used categories, with 7 and 4 use cases, respectively. These categories are used to program fully programmed robots in 7 cases. If data is available, it is much easier to make predictions for scenarios without collaborations. For training people on new scenarios, it is possible to use scenarios where there are no robots or fully programmed robots. Designing a layout is another common use case for digital twin, with 4 use cases, of which 2 involve fully autonomous robots. This is because it is easier to change a virtual layout, and it does not require a collaborative task to do it.

4.2 Extended reality interfaces for digital twin and robotics interactions: application domains, usages, and interface design

This inquiry seeks to explore the optimal extended reality interfaces for digital twin and robotics interactions, by taking into account the application domains and use-case activities (RQ2). Specifically, this inquiry aims to analyze the pros and cons of various extended reality interfaced with digital twin and robotics and how they are selected according to the application domains.

Following-up the definitions introduced by Rokhsaritalemi et al. [48], the difference between augmented reality, mixed reality, and virtual reality is seen through three key-characteristics: immersion, interaction, and information.

The chart depicted in Fig. 5 presents the prevalence of extended reality technologies. Upon analyzing the data, virtual reality is the most commonly used extended reality technology, with 23 recorded use cases. 14 use cases deal with augmented reality, and 9 use cases are referred as mixed reality. We can observe in Fig. 3 that mixed reality is not explicitly mentioned as a keyword. However, mixed reality has been included in this analysis since articles can tackle mixed reality design, development, and use cases without mentioning this designation explicitly and including it as a keyword.

The detailed analysis of the publications indicates that extended reality interface design is closely linked to application domains and usages. For monitoring-based use cases, augmented reality is commonly preferred because this technology enables the manufacturing operator and engineer to view both the real world and virtual information simultaneously without requiring any interaction. For workshop design-based use cases, a fully virtual world or a mixed world, where the operator can interact with both virtual and real objects, is preferred. Therefore, virtual reality and mixed reality technologies are respectively used in 3 and 4 use cases. For layout design-based use cases, a virtual world is required to simulate several layouts before making any real changes, and virtual reality is used in 3 use cases. For robot programming-based use cases, a virtual world is also required for testing new concepts and offline robot programming. Therefore, virtual reality is used more frequently than other extended reality interfaces. To control a robot through a digital twin, all options can be used to achieve this goal. In many cases, the data generated from the digital twin can be used for making predictions, and 2 use cases have implemented virtual reality, while 1 use case has implemented augmented reality. Finally, training-based use cases for employees can be carried out using virtual worlds or a learning-by-doing approach with augmented objects. In both cases, virtual reality is used in 1 use case, and augmented reality is used in 2 use cases.

As a summary, Table 2 provides the exhaustive list of the SLR reviewed papers with their references and gives insights about the extended reality interfaces (AR, MR, VR) attributing to each article the interfaces used, and use cases that use robots.

4.3 Digital twin concepts and variants for digital twin\(\leftrightarrow \)physical twin interactions

This section focuses on understanding the ways that a digital twin interacts with its corresponding physical counterpart. First, this section identifies the main digital twin\(\leftrightarrow \)physical twin interactions. Then, usages are clarified (RQ3).

The interactions between physical twin and digital twin can take 4 different forms [20]: digital model (DM), digital generator (DG), digital shadow (DS), and digital twin (DT). The distinction among these four forms of interaction hinges on the flow of information. If data flows automatically from the digital domain to the digital domain, it constitutes a digital twin. If data doesn’t flow automatically either from the digital domain to the physical domain or vice versa, it represents a digital model. When data flows automatically solely from the digital domain to the physical domain without a corresponding automatic return, it qualifies as a digital generator. Conversely, if data flows automatically just from the physical domain to the digital domain without an automatic return, it is categorized as a digital shadow.

Based on Fig. 6, we can observe that interactions between the physical and virtual objects occur at the level of digital twin in 16 cases. Interactions at the digital shadow level represents 17 cases. In 3 cases, there are automatic interactions from the virtual object to the physical object at the digital generator level. In 10 cases, there is no interaction established between the virtual and physical objects (Digital Model level).

Fig. 6
figure 6

Evolution from 2018 to 2022 of the digital twin concept and its variants

Full size image

As examples, Alfrink et al. [41] use digital twin concept in online robot programming. Digital shadow has been implemented for monitoring use cases as proposed by Eyre et al. [44]. Cai et al. [36] tackle the implementation of the digital generator concept for supporting a layout deployment. Finally, a digital model concept has been proposed by Weistroffer et al. [27] to simulate a factory line.

A full review of digital twin concepts and variants is depicted in Table 3.

Table 3 Matrix of interactions
Full size table
Fig. 7
figure 7

The 4 modeling pillars: geometry, physics, behavior, and rules. Boxes in green stands for modeled devices, in purple for tools and in blue for technologies

Full size image

In addition, this analysis highlights the confusion and misconceptions of the scientific community regarding the digital twin concepts: papers implementing a strict digital twin information exchanges between the real and virtual worlds represent only 35% of the articles, although this was a keyword used in the query research.

Besides the overall distribution of digital twin\(\leftrightarrow \)physical twin interaction levels, the analysis of their evolution along the time-period makes it possible to highlight the rise of some concepts and the fall of others. Figure 6 shows that the rise of digital twin interaction from 2 use cases in 2018 to 7 in 2022. Conversely, DG interaction just represents 1 use case in 2019, 2 in 2020 which indicate that such interaction have no significant added value since it is mainly used for simulation before the deployment and evaluation in real conditions.

Table 3 gives details of the interactions involved in each use cases, specifying whether they are between the physical and virtual twin (physical twin - digital twin interactions), between the human and the virtual twin (DT-Human interactions), or whether they are confined to the digital twin (digital twin-digital twin interactions). The number in square brackets indicates the number of use cases engaging this specific interaction, whereas the number in round brackets is an identifier of the specific interaction. This interaction breakdown enables the understanding on how the scientific community is implementing these interactions and where are the current limitations. Table 3 shows that most of the digital twin-human interactions engage humans in augmented reality or in virtual reality for interacting with digital robot or any other digital machines. Also, there is no teleoperation through the digital twin nor collaboration between humans through the digital twin. In addition, the table highlights that the unique type of interaction between physical twin and digital twin engages a digital robot or machine and its physical counterpart.

Analyzing the insights presented in Table 3, it becomes evident that when a human is immersed in the virtual domain through VR technology, there is a noticeable absence of interaction with entities in the physical domain. This underscores a dissimilarity between the tangible and virtual domain, highlighting that digital object representations lack the capacity to influence the physical reality. This observation also holds true for digital robots, as they are incapable of effecting any modifications within the tangible environment. Addressing this limitation requires the development of technology that facilitates the capacity for virtual representations to impact the real world, a feat achievable through the simultaneous perception and interaction with both real and virtual objects.

4.4 Manufacturing application levels and modeling pillars for digital twin and robotic construction interfaced with extended reality

This section breaks down two important components of the analysis, which are modeling pillars and application levels (RQ4).

4.4.1 Modeling pillars

In [49], Tao et al. describe a model foundation based on 4 pillars for creating complex digital twin models, which uses disciplinary knowledge from various application areas. Our review of technologies and tools are analyzed from the perspective of these 4 modeling pillars.

According to Fig. 7, Digital twin modeling consists of 4 primary pillars: geometry, physics, behavior, and rules. The green boxes depict the various elements that have been modeled, including robots, humans, and equipments. The purple boxes illustrate the tools that have been used for modeling these components. Finally, the green boxes give details about the technologies that have been used for modeling behavior and rules within the implemented use cases.

  • Geometry: various tools such as Unity, and 3D Experience were used to model industrial components like robots, drying chambers, compressors, suction devices, and heating ultra-thermostat.

  • Physics: Robot Studio module, Unity, and 3D Experience have been used to model load capacity, human arm, robot arm, joint state, and radiation wavelength.

  • Behavior: it includes the use of Robot Studio, ROS, Siemens Plant Simulation, and other tools to model operation movement, planning, pick and place, bear insert, and push.

  • Rules: open 3D libraries, Rapid Miner, TIA Portal, and other technologies like deep learning, reinforcement learning, and teleoperation machine learning were used to model cleaning, safety status, user commands, maintenance data, and assembly guides.

This comprehensive analysis demonstrates that digital twin modeling can be made through the use of a variety of tools and technologies, each catering to different aspects of the industrial applications.

We can notice that authors did not provide additional information about technologies used in physics and geometric modeling. There are two reasons for that: digital twin engines are high-level software that take care of these models; the reviewed use cases are in the industrial application domain, for which there is no added value of getting complete and exhaustive physics and geometric models. The focus is much more on the behavior and rules of the system being modeled, as these pillars have a direct impact on the overall performances and functionalities of the process.

4.4.2 Application levels

According to Lechler et al. [50], the use of a digital twin can have 4 different purposes: visualization, identification, prediction, and control.

Kuts et al. utilized digital twin to visualize warehouse data [35], while Williams et al. leveraged the interface of digital twin with extended reality to monitor in real time the level of battery of a robot [34]. Digital twin has also been used for prediction, such as in Garg et al. [29] where digital twin was used to predict the trajectory of a robot. Real components can be controlled through digital twin, as demonstrated by Kuts et al. [51].

According to Fig. 8, we can observe that the main target-function of digital twin interfaced with extended reality is “identification,” as it has the highest number of occurrences with 18 instances mentioned all over the reviewed articles. “Prediction” is the second most frequent target-function with 11 instances, while “control” and “visualization” are less frequently used, with 10 and 7 instances respectively. We can deduce that digital twin is primarily employed to determine the current state of a system. “Visualization” is the least frequently used target-function. Merely limiting the use of a digital twin to a system visualization does not add any value, since it can be implemented using a digital shadow or digital model without any interaction constraints.

Fig. 8
figure 8

Digital twin target-functions with respect to the 4 digital and physical object interactions

Full size image

It is important to avoid any bias towards the identification target-function, as the most common concept of digital twin is the digital shadow. According to the digital twin application level, the most common target-function is “control,” with 6 instances, as this is the native key function of a digital twin. Furthermore, the reason for employing digital twin for identification is demonstrated by Fig. 8. Although, implementing a full digital twin can be technologically challenging. It is easier to implement a digital shadow. Consequently, the community has devised a strategy to exploit immersive technologies while circumventing the technological barriers of a full digital twin. Therefore, in this way, a digital shadow can fulfill the role of system identification.

4.5 Software involved for extended reality integration with digital twin and communication protocols

This section aims to explore the technologies behind extended reality interfaced with digital twin and robotic systems and how they interact with the physical world. This will enable the identification of the most commonly used software and how they are used according to the application domains and use cases (RQ5). In addition, it will provide insights about communication protocols and routines which are used for synchronizing data between extended reality, digital twin, and physical twin, and how they impact the performance and reliability of the systems. Finally, this clear and comprehensive review of software and communication protocols will strengthen the awareness of companies for implementing extended reality interfaced with digital twin and robotics technologies.

Figure 9 presents the various software that have been used in the selected articles. It is evident that Unity is widely used by the scientific community, with 32 occurrences over the 46 use cases (\(\approx \)70%)

Fig. 9
figure 9

Software and communication protocols used in extended reality interfaced with digital twin technologies

Full size image
Fig. 10
figure 10

Communication protocols and sensor deployment strategies

Full size image

Figure 10 indicates that authors tend to prioritize discussing technical challenges over explaining the specific communication protocol employed. Nonetheless, some articles do mention the protocols they utilized. OPC-UA is the most frequently used protocol in the analyzed articles, with 7 instances of usage. Following closely behind are ROS Bridge, ZMQ and MQTT.

As examples, Weistroffer et al. implement an OPC-UA communication [27], whereas Shaaban et al. use a ROS bridge protocol [21]. Calandra et al. make use of ZMQ to ensure the communication [25] and Caiza et al. implement a MQTT communication [22].

To ensure reliable communication between physical twin and digital twin, there are multiple approaches for deploying a network of sensors. Sensors can be integrated into robots, manufacturing equipments, part of augmented reality devices, or deployed directly within the industrial environment without a specific device. The authors of the analyzed articles provide insights into their sensor deployment strategy, specifying whether sensors are embedded or deployed within the environment. Figure 10 indicates that out of the 46 use cases, sensors are not embedded in 15 use cases, and embedded in 12 cases. However, in 11 cases, authors did not provide information regarding the sensor deployment method.

For instance, in their work, Vidal et al. chose to follow a non-embedding approach [24], while Begout et al. embedded all sensors [23]. On the other hand, Pizzagali et al. did not provide sufficient details regarding their sensor deployment strategy [30].

Figure 10 provides evidence that OPC-UA is a widely compatible protocol with many commonly used sensors, both embedded or not. In addition, this figure shows that many communication protocols are home-made.

4.6 Challenges and limitations

This research question (RQ6) aims to investigate the various challenges about digital twin and robotics interfaced with extended reality, and provide insights about how these challenges can be overcome. Through our review, two main opportunities have been identified which could leverage the potential of this emerging technologies: data sharing; simulating dynamics scenarios; implementing generic solutions for multiple robot device programming; mixing real and virtual components in a same representation.

  • Data sharing: the platform should enable to share algorithms and data recordings, so that also remote maintenance scenarios can be explored between different stakeholders [33].

  • Simulate dynamic scenarios: by coupling augmented reality applications with a monitoring platform and various components, users can simulate dynamic scenarios and explore a wide range of system setups. They can even receive advice on suitable components for their specific use case scenarios. Furthermore, the modeling methodology can be enhanced to enable the simulation of dynamic physical interactions between objects. This upgraded process control tool can run in parallel with the actual process, leveraging extended reality technologies to retrieve important machining information and make real-time adjustments, as indicated by Stavropoulos et al. [31].

  • Generic solutions for multiple robot device programming: for companies that are exploring the use of robots in their manufacturing processes, managing the integration of multiple types of robots in digital twin and Virtual Reality environments to facilitate visualization and control can be challenging. Acquiring separate simulation software for each type of robot may not be cost-effective. One possible solution is to leverage the use of a digital twin augmented with a virtual reality system by designing generic solutions for multiple robot programming. This approach offers affordability and can be extended to automate other manufacturing processes. However, creating a customized digital twin model and immersive virtual environment requires an expert developer, which is a potential drawback. In addition, this requirement offers versatility, allowing companies to add new features and functionalities as needed [13].

  • Mixing real and virtual components in same representation: that means to integrate virtual object to the robot perception, so the real robot can take into consideration the pure virtual objects while computing its future tasks.

  • XR collaboration supported by the digital twin: as analyzed in the previous section, there is a dearth of use cases involving multi-user XR collaboration supported by the digital twin. The challenges lie both at the infrastructure level, such as the deployment of 5 G to ensure low-latency communication and near-real-time updates of the digital twin’s 3D model and semantic information, and at the software level, such as AI integration to enable object or gesture recognition, motion analysis, and seamless multi-user interactions.

5 Enhancement of physical and digital interactions: a new concept of augmented perception

The lack of interaction between physical robots and digital objects or digital humans restricts many simulation opportunities, which could be useful for programming and training purposes. Improving these interactions will increase the use of digital twin in various industrial contexts. To overcome these limitations, future researches may focus on implementing augmented perception for enhancing physical and digital interactions.

To the best of our knowledge, this new concept has not been defined in the literature yet, which leads us to introduce the first definition of its kind. To do so, we have been inspired by Rokhsaritalemi et al. [48] by making an analogy between the concept of augmented perception and augmented reality since both are close: augmented reality relies on the human point of view, whereas augmented perception deals with the robot view point. The definition is given below:

Augmented perception is an enhancement of the robot perception that requires the following characteristics:

  • Detect similarly digital and real objects,

  • Integrate these both entities in a joint representation,

  • Process them in a real time manner.

6 Conclusion

This systematic literature review conducted from 2010 to 2022 has enabled the analysis of the progress of digital twin and robotics interfaced with extended reality, which lead to know that the primary interaction between digital twin\(\leftrightarrow \)physical twin is digital shadow due to its simplicity of implementation and minimal technical constraints. The digital shadow only requires sensors to be deployed, whereas digital twin requires communication in two directions and a strong communication architecture like OPC-UA. Furthermore, the increasing use of digital twin in recent years indicates that the community is actively developing solutions that are easier to implement, unlocking the full potential of digital twin technologies that are previously limited, and inaccessible to industrial applications. The following conclusions are a summary of the previous overall findings:

  • The use of digital twins in industry is becoming increasingly prevalent, with applications in monitoring, design, simulation, programming, and training. Among the extended reality interfaces available for interacting with digital twins, virtual reality and augmented reality are the most commonly used. The choice of interfaces depends on the specific application domains and use cases. For instance, augmented reality is preferred in monitoring use cases as it allows users to see both the real and virtual worlds at the same time. Virtual reality is commonly used in design and simulation use cases, where a fully virtual world is needed to design workshops or test new configurations.

  • Despite the several digital twin concepts, the digital twin one interfaced with extended reality technologies provides the most suitable and intuitive way for interacting with physical worlds and facilitating collaboration between humans and robots.

  • Various software are used for the design and implementation of digital twins, including Unity, Gazebo, and NyAR-Toolkit. The most commonly used communication protocol between the digital and physical twin is OPC-UA.

  • While digital twin technology offers many benefits, there are also associated challenges such as simulating dynamic scenarios, integrating multiple types of robots in digital twin, and simulate scenarios where real robots interact with pure virtual objects. This concept, defined as augmented perception, will be the focus of our future works since it open up new opportunities for digital and physical interactions. Augmented perception has the potential to unlock a wide range of applications, ultimately benefiting stakeholders in the manufacturing industry by conserving valuable resources.