Keywords

1 Introduction

The terms tele-presence and tele-assistance have been evolving rapidly in the last decades thanks to technology evolution. “Telepresence” used to refers to audio and, occasionally, video interaction between a remote expert supervising a local technician in doing a specific task, while today the concept evolved implying the concept of immersive tele-presence thanks to the Augmented Reality (AR) systems.

AR tele-presence (or tele-consultation/tele-mentoring) platforms are widely and commercially diffuse in industry for maintenance purposes [1, 2], indeed AR has the capacity to deliver hands-on training where users receive visual instructions in the context of the real-world objects [3].

In the context of healthcare, from the concept of tele-consultation towards tele-mentoring and tele-surgery, AR based systems are starting to spread in literature even if commercial applications are sparse [4, 5].

Rizou et al. [6] defines a general telemedicine system as “Telemedicine is the use of electronic information and communication technologies to provide and support health care when distance separates the participants (physicians, providers, specialists and patients)”. In such a definition AR based systems seems to be the best way to provide the needed information in a telemedicine context.

A lot of existing tele-consultation platforms rely on systems that allow the mentor to overlay with AR technique graphic or textual annotations onto imagery of the environment where the proctored clinician is. These images are displayed to the trainee typically on a nearby computer monitor. It is an important issue to enhance the ability of the mentor to demonstrate proper actions of the trainee in a manageable way.

Vera et al. [7] implemented and validated an augmented reality tele-consultation platform for laparoscopic surgery, which overlaid a live view of the surgical instruments manipulated by a remotely located mentor onto the laparoscopic monitor viewed by a trainee to conduct the intervention. This approach showed the effectiveness of overlaying mentor guidance directly onto the trainee’s view of the operating field; because the trainee normally views the operating field through the laparoscopic monitor, there was no active focus shifting [7]. In fact in a very recent work Andersen et al. [4] underlined how it is important to provide to the mentor and the trainee the same point of view in order to improve the efficacy of the communication. They propose a display based platform that interpose a transparent display between the surgeon’s eyes and the patient so to offer directly in the right point of view the mentor indications.

Another interesting approach to augmented reality tele-consultation or tele-mentoring has been presented by Shenai et al., it is called Virtual Interactive Presence and Augmented Reality and uses a set of videoscopes through which both trainee and mentor could view the operating field augmented with mentor-provided overlays. This system allowed a mentor to “see what the local surgeon sees,” and was used successfully while performing a cadaveric carotid endarterectomy and a craniotomy; a major disadvantage, however, was the bulky eyewear of the apparatus that forced the trainee to operate from a fixed, rigid location [5].

Considering what abovementioned the requisite for a comprehensive and practical tele-consultation system are:

  • High-definition visualization and depth perception by both the remote and local clinician;

  • Ability of the remote clinician to “see what the local sees”;

  • Ability to provide AR facilities to both clinicians;

  • An interface that deploy standard and available internet protocol guaranteeing real time, high quality interaction.

In this respect this article presents a wearable AR-based platform that tries to address the need for immersive tele-presence and a real-time platform for tele-mentoring with the aim of provide specialist consult to low specialist remote area without the need to move the patient.

2 Methods

In this section we provide the description of the developed platform and its initial testing.

2.1 Augmented Reality Platform

The platform proposed consists of two dedicated workstations: the Local Calling Clinician (LCC) workstation and the Remote Expert Clinician (REC) workstation.

Both workstations are provided with a video see through Head Mounted Display (HMD) shown in Fig. 1. The device entails a commercial stereoscopic HMD (Sony HMZ-T2) modified with two external cameras (IDS uEye XS), one for each eye. The cameras capture real world and stream images on the internal HMD monitors so that the user feels to observe the reality with its own eyes, the internal monitors are used to display virtual information to augment the reality perceived by the user [8–10].

Fig. 1.
figure 1

Modified HMD display. The two external cameras can be moved to adjust eye convergence and field of view.

Full size image

The HMD has been developed in the mainframe of a multipurpose platform for image guided surgery, and its custom AR software libraries are used to develop the tele-mentoring platform [8].

The LCC workstation is provided by a laptop and the HMD. The clinicians, whenever he/she needs a second opinion or a mentoring, just wear the HMD, start the application and call the REC.

The REC workstation is organized as a green room (Fig. 2) for field background subtraction. In this case the visor is mounted in a fixed position and the mentor can sit and comfortably lean his/her head on the visor.

Fig. 2.
figure 2

REC Workstation. The workstation is arranged to ensure ergonomics and comfort for the operator and to maximize efficiency for background subtraction. In the red circle the HMD highlighted. Professional lighting (red arrow) to minimize shadows or chromatic aberrations for a maximum efficiency of Chroma-key, green screens highly absorbent and low reflectivity for a good result of the Chroma-key. (Color figure online)

Full size image

A key element of the system is represented by the fact that the two platforms are configured with exactly the same hardware in particular regarding the HMD and the external cameras configuration.

The two external cameras are mounted with a strategy that allows to vary camera convergence depending on the needed working distance. From a perceptive point of view camera convergence regulates the stereoscopic perception of the scene, as it allow to adjust the screen disparity of the displayed images for viewer comfort, to optimize depth perception or to otherwise enhance the stereoscopic experience [11]. In our application is important to set the camera convergence depending on the needed working distance, that can vary if the clinical scenario changes but it’s otherwise defined for each clinical scenario.

We decided to mount the cameras with a strategy to vary camera convergence in function of the application needed in a range of three different predefined positions. All the parameters are registered in a configuration file that allows to coherently adapt the convergence of the virtual cameras to consistently compose the virtual scene [12].

The fact that the two workstations share the same hardware with exactly the same camera convergence configuration automatically solves the issue related to the consistence of the two different scenes and allows to avoid the registration problem. In fact both the REC and the LCC moves their hands and instruments in the same field of view and at the same focal distance: the augmented scene is naturally aligned and intrinsically consistent in terms of focus and object dimensions.

2.2 Network and Communication Protocol

As said high quality, real-time, video streams are a fundamental technological prerequisite for the implementation of tele-medicine applications, and relies on the availability of a broadband solid network.

The aim of the project is to bring specialist support in remote areas that does not have specialist medical facilities. It is important to underline that it could be logistically unfeasible to implement a dedicated network infrastructure, so the system has to work properly with available internet facilities. Starting from the open source available libraries for developing Voice Over IP (VOIP) platforms [13–15] we decided to move towards what already used in literature to develop platforms for telemedicine [16–18].

We selected and tested four VOIP software: Skype™ (Microsoft Corporation), ooVoo™ (ooVoo LLC), VSee (Vsee Lab Inc.), Hangouts (Google inc.). The platforms were tested in terms of video resolution, bandwidth occupation and robustness of the communication. In Table 1 you can see the comparison between the identified software solutions and highlighted the selected software: VSee. VSee software revealed a secure, high-resolution video-conferencing software. The VSee video-collaboration software met all the requirements by providing extremely low bandwidth video-calling, control of bandwidth utilization with robust high resolution and strong data security features. Moreover, the Vsee company provides (commercially) the API of the software so to develop tailored platforms.

Table 1. Selected VOIP software comparison
Full size table

As for now we decided to develop the platform maintaining the Vsee software external, but in the future we intend to integrate the VOIP facilities in a unique platform.

In Fig. 3 a schematic diagram of the system is shown. The core of the platform is the QRSAR software developed on the top of the custom developed AR libraries [8]. Its role is too performs the stereo image composition both from the REC and the LCC side and build the AR scene.

Fig. 3.
figure 3

A schematic diagram of the system is shown

Full size image

More in details the QRSAR software automatically calibrate for background subtraction at the REC side and the stereoscopic frames of the segmented hands or surgical instruments moved by the mentor are composed with the LCC stream in order to provide on both side the same view: the remote mentor hands or instruments overlapped to the real local scene so to provide an immersive visual feedback that allows for an effective proctoring. In Fig. 4 an image taken during a phantom [19] test is shown that explain the afore-described process. On the right it is possible to observe the side-by-side 3D left and right eye stream once the AR scene has been composed.

Fig. 4.
figure 4

Example application of the platform in a simulated laparoscopic scenario. Left: the LCC and REC stream are showed before Chroma-key subtraction and AR composition. The remote clinicians moves in the green room a surgical instrument to guide the movements of the local clinician; Right: once calibrated both clinicians share the same view: the Augmented Reality composition of the scene.

Full size image

2.3 Preliminary Testing

Initial tests of the platform consisted in a preliminary technical evaluation of the system to qualitatively assess technical aspects such as connectivity, assessment of the video frame rate, and usability.

In particular 10 non clinicians and 2 clinicians tested the platform in three different network condition (ADSL, 4G, 3G) for a 30 min call.

The overall video frame rate ranged from 23 to 58 fps with a medium fps value of 32 fps, that means that the real-time target is reached. The video quality was fixed at 720p (HD) and maintained for the whole time of the call and the audiostream was constant.

The clinicians users report that the user interface could be improved in term of intuitivity and that the HMD could result a bit cumbersome for long calls.

Two test in two different clinical environments were conducted too.

As said our target application areas are the clinical environments, including rural medical centers. In particular it was selected a mountainous area of Tuscany, the Garfagnana, where only a generalist hospital is available; in this context a lot of patients are often required to move to more specialistic hospitals for a second opinion or for more specilistic exams.

We selected two different scenarios: an ambulatorial one, where routines control are executed that could sometimes require a second more specialistic opinion; and the orthopaedic trauma unit, where sometimes for serious trauma or fractures external fixation has to be placed before to move the patient.

In the first scenario, the ambulatorial one, we tested the system during a gynechologic control (Fig. 5), the LCC called the REC (that in this trial was simply in another room) for help during an external observation of a pregnant woman.

Fig. 5.
figure 5

Early system testing: gynecologic setup. On the right: the clinician during an exam asked for advice to an expert: On the left the expert hands guide the local clinician for effective palpation. In the center the image proposed on both the HMD is shown: the hands of the remote doctor (red circles) are superimposed perfectly and consistently to the actual scene (blue circle) and the local doctor can easily follow the instruction. (Color figure online)

Full size image

For the second scenario we tested the system with an expert clinicians guiding remotely a resident to mount an external femoral fixator: in that case we do not involve a real patient but proved the feasibility on a phantom. Such a scenario revealed very interesting in terms of possible outcome and usefulness. External fixation is a viable alternative to attain temporary rigid stabilization in patients with multiple injuries. It is rapid, causes negligible blood loss, and can be followed by subsequent nailing when the patient is stabilized. Although demonstrated benefits the mounting of an external fixator is a challenging task, especially when performed by non-expert clinicians [20]. The possibility to proctor this task through a tele-mentoring platform has been explored in literature [21, 22], and our application can reveal very useful as it allows the proctored clinicians to be effectively guided by the hands of the expert clinician in an immersive way.

Such application scenario will be further investigated and a structured clinical trial will be conducted to assess effectiveness of the platform for this task.

3 Conclusion

Our project is a step in the direction of providing specialist healthcare facilities to remote rural areas through tele-consultation. An Augmented Reality (AR) based platform for tele-health over a standard internet connection has been proposed that proved to offer a real opportunity to enhance clinical outcome.

From a technical point of view the platform revealed effective for the purposes, allowing a real-time high quality video communication and the AR application added significative effectiveness and immersive guidance.

Further testing are needed for a clinical assessment and to elaborate clinical protocols and logistic strategies for actual implementation. We also intend to explore the local impact of the telehealth service, considering the improvement of the health care delivered.