Keywords

1 Introduction to Underwater Mixed Environments

1.1 Underwater Setting as an Example of an Unconventional Environment

There has been minimal research on the general aspects of unconventional Mixed Environments [8]. In our context, conventional means that we are in a typical lab or office environment - or even outside in the landscape. In any case we have air as the surrounding medium for the display, the interaction devices and the user. An example of an unconventional Mixed Environment would be an underwater setting, where air is replaced by water. This environment creates significant challenges for conventional mixed environment technologies, such as optics, robust tracking, wireless communication, and user interaction.

There is a growing awareness that the seven seas will play a prominent role as a source of energy, food and minerals. Even the deep sea areas are already subject to intense usage by the oil and gas industry in some areas. This development leads to a number of research and business activities in sub-sea environments. As in other challenging environments, Mixed Reality could be a means to support the users via assistance, guidance or training applications.

1.2 Classification Scheme

We now present a classification scheme that allows us to distinguish different types of Underwater Mixed Environments (UWME). It is derived from analyzing the spectrum of available and future applications and uses the level of reality compared to a real-life underwater world as a second dimension. It reaches from a high degree of authenticity, i.e. the mixed reality equipment is used to mimic a typical underwater environment, over a mid level, where the real world is augmented, enhanced or maybe simplified to a minimum level, where we do not have any objects of an underwater world. The following Fig. 1 illustrates the classification scheme and gives examples for typical applications in the various categories. It is inspired by Milgrams well-known Virtuality Continuum [43] but using the specific distinction between real and virtual water on the x-axis.

Fig. 1.
figure 1

Classification scheme for Underwater Mixed Environments

Full size image

1.3 Previous Work on Virtual or Augmented Underwater Worlds

In the literature there are several examples for virtual underwater worlds designed for entertainment or education: Virtual Oceanarium [24], the SAP - Swimming Across the Pacific installation [15], the Virtual Exploration of Underwater Sites [14, 29] or the immersive virtual aquarium installation [35]. They all present a virtual underwater world that can be explored with typical VR interaction. The main focus is on a realistic experience of the scenery and specific aspects of interaction. The SAP project is the only one that reflects one important aspect of underwater environments: the lack of gravity that allows the user to float in the medium.

Similar work can be found in projects with a background in training for underwater operations in a virtual environment. The focus of the projects is set on a realistic (or at least plausible) behaviour of the technical objects or processes and a sound physical model of the operation. Typical work in this area is presented for underwater welding [62], for training of remotely operated vehicle (ROV) operators [22] or safety procedures for divers [36]. Sometimes those systems can also use real-time input of sensors and create a situational awareness for safe ROV operation [20, 26, 39]

We find a related focus in the group of VR systems that are used in the phase of product development for underwater equipment. Obviously we need a high level of physical correctness of the simulation and less visual quality. Representatives for this research work are [60] for a forward looking sonar or [61] dealing with virtual tests of autonomous underwater vehicles.

However, there are few systems actually being used in a real underwater environment. The following examples adapt Augmented Reality hardware to be used in an underwater setting for entertainment purposes: the DOLPHYN-based game [9], the AR-enriched tele-operation of a ROV [16], or the AREEF - Augmented Reality for Water-based Entertainment, Education and Fun [46].

The next section will introduce several application areas for what we call Underwater Mixed Environments. Those application areas cover a spectrum from diver assistance over astronaut training up to rehabilitation. After that we go into the technical aspects of UWME, summarize relevant physical basics and describe concrete challenges that arise from the liquid medium.

2 Driving Applications

As briefly touched upon in the first section, there are various applications that depend on Underwater Mixed Environments. We concentrate on three categories of the left column of Fig. 1 and present an important application area for each row: one biological (marine research), one industrial (ship inspection) and one medical (rehabilitation) in detail. Other useful application areas such as astronaut training, where the water simulates zero gravity are not discussed here but would lead to similar challenges.

2.1 Marine Research

Use Cases. While typical VR installations are designed for human users, there are also some examples for animal users. Empirical studies - especially in behavioral science - have been published for example with honeybees [1] or moths [27]. This kind of experiment is also useful in studying fish or other aquatic animals. For that purpose one or more displays are attached to an aquarium and camera systems are used to track and observe the animal in the underwater setting. Using the tracking information, the virtual world is updated according to the reaction of the animal. Those immersive virtual fish tank applications (from the perspective of the fish in the tank) should not be confused with the fish tank VR metaphor [63] defined as “a stereo image of a three dimensional (3D) scene viewed on a monitor using a perspective projection coupled to the head position of the observer”.

A relatively simple setup has been used to study the startle response behavior of fish according to a visual stimulus [12]. It uses one screen displaying a growing ellipse as a simplified presentation of an approaching fish. A mirror was mounted at an angle to a side of the tank, so one camera could be used to compute the 3D position of the real fish in the tank (see Fig. 2). Similar work can be found studying the larval zebra fish prey capture [57].

Fig. 2.
figure 2

Immersive VR setup for studying fish behavior [12]

Full size image

A more elaborated underwater virtual environment, the Sub Sea Holodeck, uses seven displays with 14 megapixels to produce a high resolution visual aquatic environment [32] (see Fig. 3). The setup has been used in context of a project to study how cephalopods sense, respond to, and camouflage themselves in a marine environment.

Fig. 3.
figure 3

The Sub Sea Holodeck with two different scenarios [32]

Full size image

The system can replay videos that are recorded with an omnidirectional underwater camera. This feature is used to copy real underwater scenarios to the VR environment. Alternatively, the surrounding screens and projectors can display the output of a rendering system to synthesize a controlled and reproducible visual environment.

We can summarize, that this kind of VR-enabled fish tank is a very flexible and powerful means to support research of marine biologists and neuro scientists. It allows the scientist to study the behaviour of the aquatic animals in a controlled environment and the setup can easily be instrumented with various sensors to measure the reaction.

Specific Requirements. Even though some examples use a quite elaborated model of the virtual underwater environment, most experiments show that a quite simple representation (basic shapes and changing ambient color) works quite well to stimulate the animals. While the presentation can be kept simple, the tracking of the response of the animals is the challenging part here: many experiments do not only measure the fish’s position and orientation with a high frequency sometimes even eyetracking is used to characterize the reaction of the aquatic animals.

2.2 Underwater Ship Maintenance and Inspection

Use Cases. In this subsection we discuss a list of applications, related to Underwater Ship Maintenance and Inspection (UWSMI), which seem appropriate for challenging the concept of Underwater Mixed Environments (UWME). It is just one area of possible application of Mixed Reality in the maritime industry. Additional use cases not directly linked to the underwater setting are described for example in [40] or [59].

Our view is that UWSMI can be used for boosting the research in the areas of Virtual-, Augmented- and Mixed-Reality (VAMR) environments and, eventually, improving drastically the technological status and the quality of services provided in the context of this specific application with high industrial impact.

A non-exhaustive list of UWSMI activities may include:

  • Underwater hull cleaning,

  • Propeller polishing.

  • Underwater welding.

  • Applying adhesives suitable for underwater bonding.

  • Materials underwater for both naval and commercial customers.

  • Propeller crack detection.

  • Plate thickness readings.

  • Underwater surveys in lieu of dry docking.

  • Impact damage inspection.

  • Security inspections.

  • Sea-valve inspections.

  • Oil- and liquefied-gas-terminal jetty inspections.

  • Hull-potential surveys.

The effect on the above activities in the operational and economic efficiency of a ship as well as its safety is important. For example, fuel saving is the major reason for making underwater hull cleaning an integral part of planned maintenance, especially in the era of (super)-slow-steaming. For example, marine engines manufacturer Wärtsilä [64], calculates that fuel consumption can be reduced by 58 % by reducing cargo ship speed from 27 knots to 22 knots while the large container ship Emma Maersk can save 4,000 metric tons of fuel oil on a Europe-Singapore voyage by slow steaming [34]. At a typical USD 600–700 per tonne, this works out to USD 2.4–2.8 million fuel savings on a typical one-way voyage.

In the maritime environment setting, concentration of marine fouling can lead to increased resistance, resulting in a detrimental impact on a vessel’s hydrodynamic performance and hence the relationship between speed, power performance and fuel consumption. Fouling, particularly in the case of a prolific buildup of hard or shell fouling like barnacles or tubeworm, can cause turbulence, cavitation and noise, frequently affecting the performance of, e.g., sonars, speed logs and other hull-mounted sensors. Marine fouling is considered as a global-scale problem in marine systems, costing the U.S. Navy alone $1 billion per annum [13]. Ship-hull cleaning is performed in the dockyard when the ship has her official inspection each year or by divers underwater, while the ship docs at the port. Though the regular or intermediate inspections take place each year, it is desirable to have frequent ship hull cleaning in order to keep good fuel efficiency for securing lower transportation costs and \(CO_{2}\) emission.

Analogously, the operational need for polishing a propeller stems from the fact that a super smooth surface is inhospitable to marine organisms as well as being beneficial to the efficiency of the propeller. In this connection, provided the service is done properly, the cost of the underwater propeller polish will only be a small proportion of the fuel savings.

UWME could be used for training divers in nearly realistic conditions by means of augmentation in their diving mask. Such an environment could be based on a towing-tank-like facility endowed with immersed physical mock-ups of full-scale parts of the ship hull and, if necessary, its propeller blades and appendages, as well as an AR interface capable to superimpose on them computer generated or photographic images. Such an installation would readily inherit from its physical counterpart, the surrounding medium and the effects of buoyancy, viscosity and free-surface waves while offering the trainees with a realistic perception of their working environment regarding its lighting (refraction, absorption and scattering by the water particles), water turbidity, spatial complexity and limitations (hull stern with its propeller), and the activities to be performed on it. One could bargain limited perception losses, e.g., buoyancy in fresh versus sea water, in favor of effective wireless communication, since it is known (see, e.g. [33]) that conventional RF propagation works poorly in sea water due to losses caused by its high conductivity (typically 4 S/m) versus that of fresh water (0.01 S/m).

In addition, UWME could be useful in the context of design, construction, testing and approval of mechanisms devised by the industry for supporting ship-hull maintenance and inspection [45].

On the basis of the above and the fact that VAMR technologies have been already acknowledged as a useful mediator for challenging maintenance services in maritime applications (see, e.g. [38, 58]), one could reasonably expect that the current and emerging industrial needs should have already led to coordinated actions for testing the feasibility and efficiency of UWME for supporting UWSMI. Nevertheless, searching the pertinent literature the reader gets the impression that research in the area of UWME-for-UWSMI, with the exception of scarce high-quality attempts (e.g., [44]) has not yet grasped the threshold for being characterized as an emerging research area, even more, an emerging technological area.

Specific Requirements. For offering a realistic training environment, UWME should be enhanced with additional functionality regarding its physical components, enabling, e.g., the generation of currents, fluid rotation, turbulence effects, etc. Furthermore, the AR interface could be enriched with haptic devices for providing the user with a tactile feedback for improving the degree of realistic perception of activities that involve the operation of devices controlled by the user and acting on the underwater part of the ship hull and its appendages, e.g., operating a cleaning machine at the proper pressure level for removing mild or persistent fouling from a variety of coatings, including the new low-surface- energy coatings being introduced into service.

Furthermore, the following requirements can be derived from those industrial use cases for UWME:

  • The systems must be robust enough to be used in the harsh environment of professional divers.

  • The alignment of virtual objects and real objects (in an AR setup) must be highly accurate.

  • The training environment should be easily adaptable to the concrete task and the necessary tools in order to cover a broad range of use cases.

2.3 Rehabilitation

Another of the main driving applications for UWMEs is rehabilitation. In this application there is a significant need for Virtual Environments (VEs) that work underwater. Water-based exercise is one of the primary physical therapist recommended rehabilitation approaches for many types of injuries and disabilities.

Background

Water-based Physical Therapy. Water-based (aka aquatic) Physical Therapy (WPT) has been used for many years to rehabilitate individuals with a variety of disabling problems. WPT utilizes the physical properties of water, such as providing resistance when moving through it. These properties enable a wide range of positive effects in therapy and exercise. The water offers buoyancy and hydrostatic pressure to provide additional support for the patient when performing the exercises and reduces the risk of falls. Thus, aquatic therapy and exercise can be of great benefit to balance impaired populations, such as the elderly [19]. Moreover, water can aid in the body’s cooling process, which is especially important for MS patients [25], who commonly suffer from an exacerbation of symptoms due to overheating. Similarly, WPT has been used for many years to benefit individuals with other neurological problems, such as Parkinson’s Disease [65].

Virtual Rehabilitation. Research suggests that VR and AR technology can offer new opportunities and methods for neurorehabilitation [17]. Typically VR or AR can be used to immerse a patient in a safe environment to practice rehabilitation exercises [54], such as hand [42, 49] and motor [50] rehabilitation in stroke and gait rehabilitation in Parkinson’s Disease [47].

For example in MS rehabilitation, rhythmic audio signals and a moving checker patterned floor [5] can be used effectively for feedback. Other researchers [53] focused on haptics for upper extremity rehabilitation in MS. These approaches had only preliminary evaluations and have never made it to market, but they were shown to improve movement while in use. This suggests that VR approaches could be effective in gait rehabilitation for MS, but none of the prior work has developed VR systems that could be used in conjunction with water-based therapies.

Rehabilitation Games. VR Rehabilitation games have not been completely incorporated into common therapy practice, but they do seem to have significant benefits to rehabilitation. For example, a VE is not subjected to the dangers and limitations of the real world [10, 18], which expands the types of exercises that patients can practice, while still having fun. In general, research suggests that VR games have measurable benefits for rehabilitation effectiveness [18] and motivation [10].

There has been recent research on deriving design guidelines for VR rehabilitation games based on results of empirical studies. Alankus et al.’s [2] guidelines include: simple games should support multiple methods of user input, calibrate through example motions, ensure that users’ motions cover their full range, detect compensatory motion, and let therapists determine difficulty. There is a need for more focused game design research and development for specific populations [23].

User Descriptions. In the context of rehabilitation we have to distinguish between two important user groups that have different roles in the rehabilitation sessions.

Patients: Since water-based exercise is recommended for many different injuries/disabilities, the potential patient population for this application is widely diverse. Here, we will consider an example population - multiple sclerosis (MS) patients. MS is a degenerative neurological disease that affects 400,000 people in the U.S. and over 2.1 million worldwide [52].The most common form of MS is relapsing-remitting, in which patients experience acute attacks followed by periods of remission. During these remission periods, physical therapy has been shown to most effective in the remission periods and can help counter the residual effects of the attacks [51]. Therapists often augment the diminished proprioceptive feedback with other modalities of feedback, such as using mirrors to provide visual feedback [37].

Balance and gait (i.e. walking patterns) abnormality is one of the most prevalent symptoms that MS patients experience with 85 % of patients complaining of gait and balance [4]. Gait problems are exacerbated by other MS related symptoms such as fatigue, decreased reaction time, and attention deficits, especially in dual tasks such as walking while talking [28]. One of the most common issues experienced in MS is exacerbation of all MS symptoms when body temperature is increased. This makes rehabilitation and exercise significantly more difficult and reduces the benefits. To counter this overheating effect, therapists generally recommend water-based exercise, because it offers effective resistance training while keeping the body cool. It also reduces the risk of fall due to balance deficits, which are very common in MS.

Physical Therapist: For water-based rehabilitation, physical therapists are frequently present to help guide the patient and maximize the efficacy of the workout, especially at the beginning of a rehabilitation program. A physical therapist first creates a personalized set of exercises based upon the needs of a patient. Then they teach the patient to perform these exercises correctly, offering visual, auditory, and haptic feedback to the patients as needed. Therapists often rely both on patient history and visual assessment of the patients movement to drive their feedback. Thus, being able to visually assess patients in real time is critical for a physical therapist to effectively guide the patient and provide additional motivation.

Use Case. This use case considers a therapy session in which both the physical therapist and the patient are working together in the pool. The patient may be playing a game a rehabilitation game in a head mounted display, and is experiencing many virtual stimuli. To integrate exercise, the patient runs back and forth along the length of a lane while carrying water weights - large dumbbell shaped, air-filled objects that provide additional resistance when pushed through the water. The therapists aim is to assess the movements of the patient and provide corrective feedback through multiple modalities: verbal, visual, and haptic. Moreover, the therapist may also want to control some of the game events and difficulty level in order to tailor the game to the individual patients needs.

The therapist may need to be immersed and embodied inside the VE to provide effective multimodal feedback to the patient (i.e., otherwise the therapist would be unaware of what the patient was actually seeing). To facilitate this, the therapist will have a tracked egocentric view similar to the patient, but will also have a heads up display (HUD) that can provide additional information to the therapist. For example, the therapist can monitor the patients physiological data, such as heart rate. The periphery of the display may need to be unblocked due to the therapist’s need to interact with the real world. For example, the therapist may wish to observe and correct the patients real movements outside of the VE. Moreover, depending on the patients progress and frustration level, the therapist may need to control the difficulty of the game or trigger events (e.g., boss fights).

Specific Requirements

Safety Requirements One of the primary concerns of any VE system for rehabilitation is user safety, which is arguably even more important in an underwater system. For a population with disabilities, who may have issues such as balance deficits or vertigo, there is an increased chance of falls, even in water. Considering the immersiveness of the system, the user may be surprised when they fall and their head is submerged in the water, which could result in an increased chance of aspirating water. Moreover, due to users potential vestibular deficits, users may have difficulty reorienting themselves to above the water line. Thus, the user may need additional visual and/or auditory feedback to help them reorient themselves. In general, it is advised for someone to follow behind or beside the user to ensure their safety and to help prevent falls. Mobile harnesses such as those used in gait rehabilitation, could be used, but they will need to be specifically designed for underwater usage or otherwise they could make fall recovery actually more difficult.

Whenever electronics are submerged in water, there is always the safety concern of electrocution. Thus, it is advisable to waterproof powered equipment and minimize active instrumentation of the user. For example, for optical tracking of the user, passive tracking markers or markerless tracking will be the safer choice because there is no chance of electrocution.

Real Environment Requirements. One potential requirement for the real environment - the pool - is its depth. The user should be able to stand in the pool with their head above water. Deeper pools would increase the danger caused by underwater disorientation if the users could not feel the bottom of the pool. Fortunately most gym pools are designed to be about 4 or 5 ft deep. If only a deep pool is available, the proposed UWME could be combined with a traditional pool lift - a powered mechanical arm with a seat on the end, which is strapped to users to safely lift them in and out of the pool.

Display Requirement

  1. 1.

    If a head mounted display is used, obviously it must be waterproof.

  2. 2.

    If the user is required to use the length of the pool, it should be untethered so that it does not interfere with the users movement. However, another potential approach is to use an endless pool, which is analogous to a treadmill in air-based exercise. The endless pool is small 15 ft pool with a wave generator on one side and sometimes a waterproof treadmill on the floor. This would enable them to run in place in one direction while they appear to be moving in the VE. The VE as described in the Serpents treasure use case would have to be changed to enable the user to always move in the same direction (e.g., one long hallway), unless the endless pool was modified with multiple wave generators, analogous to an omnidirectional treadmill. Another option is to use the aforementioned pool lift, which hypothetically could be modified to control orientation as well.

  3. 3.

    While not a strict requirement, it may be beneficial for the display to be see through. The purpose of this would be to give users more feedback about the water level, which would enable them to reorient themselves more effectively in the event of a fall. In many cases, there may be a trade-off between safety and the immersiveness of the system.

Interface Requirements

  1. 1.

    All interfaces must be waterproof.

  2. 2.

    Interfaces should not interfere with movement, unless intentionally designed for that purpose. E.g., it would be detrimental if an interface became tangled with the users legs. On the other hand, if the interface was integrated with the water weights, for example, it would technically interfere with movement, but for the purpose of increased resistance training.

  3. 3.

    Standard interfaces with buttons should be affixed to the user. If the user interacts with a handheld device and drops it, it would be potentially difficult to retrieve if it fell to the bottom of the pool or floated away.

3 Research Challenges

After looking in quite different usage areas we will now derive the research challenges that arise from those novel applications. There we distinguish between the technical challenges of the underwater setting and the human factor aspects.

3.1 Technical Challenges

Optics. With regard to the display of close to reality virtual underwater environments on screen, several aspects have to be considered, which strongly differ from conditions in air, especially optics and visibility. An additional aspect to be discussed is characteristic illumination artifacts.

The first aspect is the occurrence of refractive effects due to the different propagation speed of light in water and in air. This property of water can be best described by its refractive index. Water has to be considered as a transparent object in the rendering pipeline, altering the direction of light rays emanating from illuminated objects inside the water body.

In the following, the three main viewer locations are presented with regard to refraction. These comprise a viewer looking into water from outside the water body, vision in water and a viewer looking from inside the water body to above the surface. A light ray is refracted at every boundary between participating media with different refractive indexes on its way. This can be a direct contact of the viewer’s eyeball with water or one of the more common cases of looking through the surface of the water itself or any kind of water-glass-air transition. The glass can be a waterproof protecting surface like a side of an aquarium or a divers diving goggles. As the refractive index of air is close to zero and the glass is optional and mostly thin, the most significant quantity is the refractive index of water. It is commonly known to be close to 1.33.

The computation of refraction can differ severely due to the complexity of the respective surface. Planar surfaces can be represented by a single surface normal. A planar refractive surface always leads to non-linear distortions with increasing incident angles towards this normal. The computation of close to reality distortions gets even severe if one is looking through a naturally wavy water surface. Such a more complex surface cannot be represented by a single surface normal for refraction computation. A commonly realizable refractive effect of water is a magnification of objects. This magnification is a result of objects being seen closer to the observer as they are in reality. For perspective projection this means that points on a virtual object, non-ambiguously related to points on the real object, seem to be mapped (see Fig. 4). This virtual location of object points is proposed to lie exactly on the refractive surface’s normal through the real object point as partly stated in [7] and experimentally confirmed by [21].

Fig. 4.
figure 4

Refraction of a light ray emanating from object point O on its way into the camera and the corresponding virtual point V (Color figure online)

Full size image

A special phenomenon called Snell’s window or optical manhole occurs, when the viewer is looking from inside the water body to above the surface [41]. In contrast to the aforementioned cases, the refractive index gets bigger at the transition from air to water. Hence, light rays get compressed to the circular region of a bottom of a view cone. Regions on the outside of this view cone are either dark or recognizable reflections of the underwater environment.

The second aspect differing from conditions in air is visibility underwater. Its peculiarity is comparable to haze in air with an additional characteristic color cast. The color cast is a result of the water’s capability to absorb light wavelength-dependent. While longer wavelengths of visible light (red to yellow) are absorbed after reaching only a few meters in water, the shorter wavelengths (green to blue) penetrate the water body the most. This leads to the typical water color. Differences amount due to the local constituents of water. Besides the absorption capabilities of water itself, local constituents, with their own absorption capabilities, contribute to the overall absorption of the water body. Hence, locally different color casts arise, like the very blue water color in the Red Sea and on the contrary, the green color of the Baltic Sea. Some organic constituents lead to an observable, continuous flow called marine snow.

This restricted penetration of light in water is called attenuation. Besides absorption, attenuation is influenced by the scattering capabilities of the water body. It comprises diffraction and refraction. Scattering can be further divided into forward scatter and backscatter [31]. Its effects can be compared to using car headlights in fog. Following [31], image formation is a process of a linear superposition of backscatter, forward scatter and a direct component reaching the imaged objects from the light source. This model for image formation, or any other model considering all the aforementioned factors influencing underwater visibility, has to be taken as a basis for rendering close to reality underwater images.

The last aspect representing a difference to conditions in air is the appearance of characteristic illumination artifacts. Wavy water surfaces can lead to light refractions that converge and result in so called sunlight flicker on the bottom of the sea. These are bright light patterns in an irregular and fast varying arrangement.

Obviously, there are various factors that heavily influence the perception of images in underwater settings. We do not only have to take this in consideration for presenting virtual objects to the user but also when analyzing or interpreting underwater images - as we need it for example in the case of optical tracking.

Wireless Communication. In conventional virtual environments we find a variety of interaction concepts, such as gesture based interaction, flight sticks, Personal Interaction Panel or mobile devices (e.g., smartphones) for navigation and manipulation in the virtual world. Most of the techniques rely on 6 degrees of freedom (DOF) tracking of devices as an essential part of picking objects or describe gestures. In underwater settings, electromagnetic tracking will not work at all and optical tracking is much more difficult, as described in the previous section. Furthermore, Bluetooth will not work and WiFi signal strength is decreased due to the attenuation of the radio waves. One can easily verify this with a waterproof smart phone.

Tracking. Besides the limitations in wireless communication, we also have the problem of tracking. The positions of the Global Positioning System cannot be used and magnetic tracking systems may not work effectively. So we can only rely on optical or acoustic systems for tracking the user - with all the drawbacks of their robustness, fidelity and latency in underwater settings. Especially the robustness against changing light conditions remains a challenge for practical use in outdoor settings [46].

Harsh Conditions. If we put technical equipment underwater, we have to deal with some obvious technical challenges:

  • We have to carefully shield the sensitive parts (especially all electric and electronic components) against the water. Equipment that is protected against the effects of continuous immersion in water is classified as International Protection Marking IPX8.

  • According to the depth of the water, we have to safeguard the technical systems from the growing pressure.

  • Equipment used in salty water has to be protected against corrosion.

  • All materials that are exposed to (natural) water for a longer period will be subject to marine biofouling. This will not be relevant for most of the parts of the equipment but affects the optical parts such as displays or cameras.

3.2 Human Factors Challenges

Usability. Although usability guidelines for graphical user interfaces are very well defined and 3D user interfaces and VRs design guidelines are becoming more well defined, the usability of VEs underwater is not well understood. It is not clear if many of the guidelines used in air-based environments will still enable usable interfaces in an underwater environment. Consider that the usability of air-based VEs has been studied for over 30 years and is still being investigated as new interface and display technology is developed. Thus, the challenge here is the amount of empirical work it will take to derive usability guidelines for underwater VE interface design.

Ergonomics. As far as we know, there have not been any study of ergonomics for underwater applications of Mixed Reality so far. Similar to conventional MR environments there are several challenges when we try to (partially) replace the real world by a virtual world that does not have the same resolution and consistent behaviour. Due to the fact that especially the tracking problem is much harder to solve in an UWME and that wireless communication typically has more latency will make those studies even more important.

Relative to the study of healthy users in VEs, users with disabilities have always been significantly understudied. This is largely due to the variable nature of disabilities and the limited numbers of users with disabilities, making it difficult to conduct studies with a homogeneous population at an acceptable sample size. Moreover, it is unclear how users with disabilities would interact with an underwater VE. We already know that disabled persons may experience presence differently in an air-based VE. Thus, even if it was known how healthy persons interact with underwater VEs and we had derived the associated usability guidelines, it would be of minimal help towards understanding how many users with disabilities interact with VEs underwater. That is, the wide variability of disabilities will be a difficult challenge to overcome.

An additional challenge has been that VEs have traditionally been large installations, requiring the users with disabilities to come to a VE lab or rehabilitation clinic to use the equipment. This has begun to change with the advent of inexpensive VE hardware, such as the Microsoft Kinect and the Oculus Rift. However, with underwater VEs, regardless of the advances in hardware, most people do not have a pool in their backyard, which will require them to go to a local gym where the equipment can be used.

In a sense, the underwater VEs could be difficult to access installations, much like the high end VEs have been, thereby limiting their usage and study. Thus, the study of underwater VEs will be challenging, due to the logistical issues with being co-located with a pool.

4 Towards Implementation of UWME

4.1 Available Building Blocks

Controlling Color Cast. Depending on the setup and the application, we have the need to correct or simulate water-induced color cast in context of a UWME. This allows us to generate a realistic impression or improve the quality of acquired images, e.g. for optical tracking.

There are several algorithms to correct the color cast induced by the attenuation of the light depending on the different wave lengths. They go beyond a simple white balance and take into account the physical behaviour of light in underwater settings [30].

There a two possible solutions to integrate color cast correction in an UWME. First, it can be used as an amplifier for the display module. With a knowledge of the distance between underwater display and the user and some parameters of the medium we can especially raise the red and yellow frequencies. Second it could be used at the receptor side - either for a human being or a tracking system. Here we have to restore the attenuated frequencies of the spectrum to restore the original color distribution.

Distortion Correction. Distortion correction for underwater images differs severely from distortion correction in air. Distortion in air is a result of non-ideal projection capabilities of real lenses. It is dependent on the type of lens used, ranging from wide-angle, normal, to long-focus, all producing different amounts of distortion. Another factor is the quality of manufacturing of the single lens or respectively the lens-system. Distortion is most noticeable as radial effect producing pincushion to barrel distortions.

Besides radial distortion of real lenses, distortion in underwater imaging is additionally affected by a refractive interface. Light rays are refracted on their way into the camera (water-glass-air transition). This leads to noticeable distortions. The most common refractive interfaces are flat or domed ports. Flat interfaces lead to effects like radial distortion and the different refractive indexes of the participating media lead to an unintended magnification of objects in water. Domed ports are supposed to eliminate effects of flat interfaces, leaving just the magnification problem. The possible combination of lens type and interface type results in different non-linear distortions in underwater imaging.

Distortion underwater is additionally affected by the position of the camera to the refractive interface. For distortion correction, it is needed to compute the respective incidence angle of every pixel’s ray. When this is known, a mapping with refractive distortion compensation should be possible. This mapping has to be a combination with in air distortion correction from camera calibration.

Underwater Communication. Wireless communication based on high frequency radio waves will not bridge more than 25 cm underwater due to the strong attenuation of the waves [48]. This means that wireless connections such as WiFi or Bluetooth will practically not work in an underwater setting. Acoustic waves have a long range in water but suffer from high latency and poor bandwidth. As described in [55] a short range underwater acoustic channel can only transfer 20–50 kbps with a latency of \(\sim \)300 ms. Furthermore, acoustic communication modems are error-prone in shallow waters. For usage in the context of coupling an interaction device with the Mixed reality system this would not be a good choice.

This makes optical connections the better choice. Although limited to short distances and depending on the turbidity of the water, especially in blue/green light wavelengths, offer an adequate alternative [3]. The light propagates much faster and by this avoids the high latencies. However, the optical communication needs a direct line of sight which cannot be assured in interactive scenarios.

Physical Protection. Protecting electrical, electronic or optical components for underwater operation is a standard offer of specialized suppliers. The traditional way of using waterproof housings for all the equipment to be used underwater is now complemented by the concept of pressure neutral systems, where the components are embedded in silicon. First research for this approach has been done in the 1970s [6] and it has a revival now for the lightweight design of underwater vehicles [56]. However, those housings make the technical equipment more expensive and sometimes difficult to handle.

4.2 Future Research

As already discussed, we see plenty of useful application areas for UWME. However, to exploit the potential of UWME for marine research, training, assistance and rehabilitation, we need further research in the following areas:

  • Human factor research in UWME.

  • Usability of devices for underwater usage (displays, interaction etc.) that take into account diving equipment and also people with disabilities.

  • Fast and accurate underwater tracking.

  • Reducing latency in underwater wireless communication.

  • Robust underwater equipment to set up UWME.

  • Systematic approaches for design, test and operation of UWME.

  • Reusable building blocks for fast implementation of UWME.

In order to develop solutions that work in practice, it is absolutely necessary to form interdisciplinary teams that combine the expertise of visual computing, the specific application area and underwater technology.