Keywords

1 Introduction

New forms of ‘realities’ have emerged. ‘Reality’ technologies such as Mixed Reality, Augmented Reality, Virtual Reality and Augmented Virtuality are increasingly becoming indispensable to industry. Ranging from traditional application domains of architecture, design, and learning; these novel realities are the cornerstones for the this digitalization prone and smart-centric century: in a nutshell, these realities replace or merge with the normal physical world and subsequently be molded to enhance specific design comprehension, collaborating activities [7], and visualization activities [10]. These realities though promising on many fronts (industry wise) and having the ability to integrate seamlessly with the IoT revolution pose new challenges to the inherent aspects of decision making, integration of newer social-cultural and cognition capabilities (here recalibration) and influences. Clark and Chambers [2] in their ground-breaking article ‘The Extended Mind’, showcase the human cognition has the behavioral ability to realize familiar mental states and physical skills in structures and processes located beyond the scope of their immediate sense modalities; and the complete realization of a situation requires various forms and tendencies of external scaffolding and support (stimulus) oriented mechanisms. Involving user centric mixed reality in this context is a calculated extension, and as Doswell and Skinner [4] opine that ‘scientifically-grounded methods for identifying appropriate information presentation, user input, and feedback modalities in order to optimize performance and mitigate cognitive overload’ is a challenge for Human-computer interaction ecosystems, and similar is the case with AR and other forms of MR which base their ecosystem on delivering maximum user interaction.

There are three important features for any MR system:

  1. (1)

    combining the real-world object and the virtual object;

  2. (2)

    interacting in real-time; and

  3. (3)

    mapping between the virtual object and the real object to create interactions between them [6].

The main goal of MR is the creation of a big space by merging real and virtual environments wherein real and virtual objects coexist and interact in real-time for user scenarios [1]. This focus of user immersion for the output of productive decision making in ICT trends today requires appropriate visualization strategies that take into account the ‘extended’ capabilities of a human user and this informs -the design of the application cognitively.

For the purposes of this paper, we showcase the initial steps of integrating adaptive cognition elements (user centric) and expound our theoretical hypothesis covering Design and Cognitive factors to be considered while designing an effective Augmented/Mixed Reality system solution for an offshore oil rig maintenance scenario. This use case involves critical tasks in the areas of equipment maintenance, quick error recognition, safety, effectiveness etc. This involves multiple stakeholders. With the goal towards enhanced effective Situational Awareness of the Oil Rig industrial system, the experimental setup and prototypes designed would be elaborated. The paper extends towards identifying the key basic Cognitive and Design principles that designers need to consider while designing an industrial experience such as this.

2 Situational Awareness and Its Requisites for Sense Making

Situation Awareness (SA) of the user is of prime importance for an integrated and productive industrial workflow. With the scenario showcasing an OIM maintaining an offshore oil rig (from the mainland on an extended scenario), a stabilized depiction of tailored situational inputs becomes the key. Highlighting this journey of SA and incorporating Endsley’s initial theoretical framework of SA [5] and involving the Lundberg’s holistic visuo-spatial centric enhancement [9], we have incorporated AR/MR integration for enhancement of an OIM’s task in the sense of making it more streamlined, direct and most importantly the possibility of him being productive on the go. Endsley’s framework consists of three interconnected parameters:

  1. a.

    Perception of elements in the current situation within a volume of time and space

  2. b.

    Comprehension of their meaning

  3. c.

    Projection of future status

Importantly, the three conducive parameters are affected by visual display support and automation aids.

Lundberg [8] enhances on this by incorporating specific visual artifacts that act as stimulus input for the user and is of the opinion that the visuo-spatial input is ‘anchored in the present’ but performs as the ‘locus of decision-making’ to incorporate the past scenarios and proceed towards a future action in the present midst of an ‘ongoing, dynamic, uncertain process’. This locus that he points out rests in the sensemaking and the data driven frame that the user (in our case) is exposed to: this centrality of framing data is resting on the correct symbiosis (again this has to be generated by the OIM) being deduced and affected upon. Critically, sense making opens up the gaps of requisite imagination [11] and requisite interpretation during the comprehension of the present situation [8]: both of which are inherent problematic concerns for a high-stress positionality of an OIM’s work output. Taking this crucial factor of mitigating the sense-making gap, our approach is detailed out in the next section of the paper which highlights an oil-rig maintenance use case.

3 Application Use-Case and Generation of Cognitive Prototype Design

The application use case is Oil-Rig motor maintenance and the persona defined is an Offshore Installation manager (OIM). The key tasks of an OIM are monitoring, updating and allocating resources, delegating tasks to maintenance engineers (ME), collaboration and communication, implementing the plan and assessment of the situation given the information. In an oil rig, while the OIM can benefit from digital information, there is also an increased need for the OIM to remain in contact with physical reality when using several sophisticated tools. Nevertheless, the OIM is also responsible for recognizing the signs of stress, ensuring that all staff are regularly updated on situation along with demonstrate flexibility and adaptability as plan of action and goals change.

Moreover, there is not just one but too many factors (visual elements) within the person’s environment that constantly compete for visual awareness and control of action. In such an overwhelming situation information overload is bound to happen. To tackle with this problem of information overload, our attention system actively and rapidly prioritizes and selects information according to its relevance for processing and achieving the goal. But one cannot deny the fact that detection of the relevant information while ignoring the irrelevant one might get affected by the increased cognitive load on the system. Therefore, in oil rig having knowledge of objects, situations, or locations in the environment is not only useful but even life-critical for workers as they may require assistance with (1) dangers, obstacles, or situations requiring attention; (2) visual search; (3) task sequencing; and (4) spatial navigation.

Some situations would involve instances where responses are novel, dangerous and technically difficult while a few would involve planning and informed decision making, need for troubleshooting and error correction. Others may require a need to overcome some strong habitual response. But in most of the cases the most important thing that is common for all the activities is rapid generation of behavior in response to a new situation.

Hence, the design of the AR/MR system had to be as such to enable the detection of inconsistencies for OIM in the easiest manner possible for effective and efficient decision making.

The goal was to streamline the process of performing tasks where consideration is given to the related integration of cognitive functions while sparing OIM from the functional cues that are of no immediate need. Hence, the primary goal was addressing the human needs and how to augment human capabilities to overcome cognitive limitations in a decision-making process. The secondary goal was to make the system smarter as a result of the primary goal.

The prototype design defines a specific OIM workflow where the steps in the process of motor maintenance are listed as follows –

  1. 1.

    OIM is currently working on a task using Hololens. He is focused on his current task of analyzing a Large Drive as a part of his daily routine.

  2. 2.

    OIM attention is directed to an emergency. A visual cue is provided in his periphery indicating an emergency.

AR suffers a major setback that hinders the complete realization of its potential – The field-of-view (FoV) of the AR hardware is severely lacking as compared to the human eye, or even VR. With this context, the design was particularly perceptive to the aspects of the applications that focused on subtle, yet fairly accurate cues which fulfilled their purpose of guiding the user’s attention to certain elements in the application environment.

  1. 3.

    OIM uses voice command to attend to the emergency.

Spatial tracking allows OIM to interact through pointing and natural voice commands.

  1. 4.

    The system opens OIM main dashboard with the error card details. Error information and rectification to enable the detection of inconsistencies for OIM in the easiest manner possible for effective and efficient decision making (Fig. 1).

    Fig. 1.
    figure 1

    Showcases steps 1–4.

    Full size image
  2. 5.

    OIM chooses to deep dive into the task.

The system by allowing selection gives OIM a perceived sense of control.

  1. 6.

    A confirmation message is popped up checking if the OIM would want to leave the current task and take up the new one.

System confirming the action to be taken.

  1. 7.

    a) The OIM confirms to take up the task and is asked to place the 3D model in a location wherever convenient. - System allowing comfortable navigating and move around easily.

    b) The model is placed and the error details is showcased. The physical error location on the Oil Rig is also highlighted.

    c) Spatial tracking to allow OIM to interact and have the perception of ‘Where I am’.

    d) OIM clicks on the object to deep dive to next and consecutive level. Presentation of relevant information while ignoring the irrelevant ones.

    e) OIM is at the final stage with details of error information.

System is in a ready state for the OIM to delegate tasks to the Maintenance engineer. The maintenance engineer then gets instructions in their HoloLens to work on an existing maintenance task at hand (Fig. 2).

Fig. 2.
figure 2

Steps 5 to 7 have been highlighted.

Full size image

4 Conclusion

The system has been designed in such a manner that it addresses the OIMs needs and augments capabilities to overcome cognitive limitations in his/her decision-making process. It also considers the following three aspects that has not yet been well-developed from an industrial experience point of view: First, not everything in the physical world is interactable or tagged. Second, the users still do not have well developed mental models of AR applications that might exist. Third, the visual language that these AR applications need to follow has still not reached its potential.

Incorporating of the above mentioned cognitive centric aspects help to give designers a perspective on how to evolve the visual language and develop an application that attempts to deliver the best in class visual immersive experience. The observed cues are along 3 dimensions – the goal to be accomplished (task), the extent to which the cue blends in with the environment (markedness), and the context in which the cue is triggered into the user’s visibility (trigger) within the context. The challenge lies in keeping these cues minimalistic yet informative enough, so that the OIM remain immersive enough and the information is easy to understand and distinguishable from the environment. The result of this activity would be a smarter system that effectively incorporates expectations of the physical environment into the AR application and parallelly develops a well-defined visual language for the AR systems in an industrial setup.

However, our immediate next step which is already in process is the scientific validation of the prototype with actual users. Verification and validation with data from eye tracker and user-specific tasks activity will focus on the following:

  1. 1.

    Validating with the actual users

  2. 2.

    Quantitative validation - latency and accuracy

  3. 3.

    Scalability of the framework to additional industrial use cases and domains.

Additionally, we shall also focus on developing a more robust and texture specialized Visuo-Haptic Mixed Reality layer [3], as a continuation of the work mentioned in this paper.