VR-Assisted Physical Rehabilitation: Adapting to the Needs of Therapists and Patients

Abstract

Virtual Reality technologies are slated to transform the practice of physical rehabilitation and the potential benefits have only started to be explored. We present in this paper a direct motion demonstration approach for allowing therapists to intuitively create and edit customized exercises and therapy programs that are responsive to the needs of their patients. We propose adaptive exercise models, motion processing algorithms, and delivery techniques designed to achieve exercises that effectively respond to physical limitations and recovery rates of individual patients. Remote networked solutions are also presented for allowing therapists and patients to intuitively share their motions during real-time collaborative therapy sessions. Our solutions have been implemented as a low-cost portable system based on a Kinect sensor, and as a high-end virtual reality system providing full-scale immersion. We analyze and discuss our methods and systems in light of feedback received from therapists.

1 Introduction

Physical therapy is a broad field that addresses the recovery and treatment of injuries, physical impairments, disabilities, diseases and disorders related to motor and balance dysfunctions affecting many daily life activities. A rehabilitation process is usually necessary for patients after a specific type of injury involving physical (impingement, surgery, arthritis, etc.) or neurological (strokes, neuropathies, etc.) impairments.

Rehabilitation and physical therapy are optimal when assessment, monitoring, patient engagement, and adherence to the therapy program can be achieved. Different processes are involved: physical examination, evaluation, assessment, therapy intervention, monitoring, and modification of the therapy program according to patient recovery [4]. In traditional physical therapy, after a preliminary step of diagnostic and quantitative measurement a patient is guided by a trained therapist to perform specific therapeutic exercises. The tasks performed are designed according to a recovery plan, which implies repetitions of exercises and constant progress evaluation both qualitatively and quantitatively.

The process is usually intensive, time consuming and dependent on the expertise of the therapist. It also implies the collaboration of the patient who is usually asked to perform the therapy program multiple times at home with no supervision [2, 37]. Patients often perceive the tasks as repetitive and non-engaging, consequently reducing their level of involvement [20, 24]. This fact is related to a number of aspects: lack of customization on how to execute exercises, communication and interaction practices that are unsuitable to a particular patient, no clear perception of improvement, lack of coaching and monitoring while at home, etc. Addressing these many aspects is also important to improve therapy outcomes, and in particular to reduce the risk of injuries due to wrongly executed exercises.

While it is clear that VR-based computer systems for therapy delivery have great potential to well address most of these issues, implementing effective solutions involves multiple challenges. In any case, current practices can be certainly improved. For example, Fig. 1 illustrates how exercises are typically described in paper to patients when they are given a set of exercises to be executed at home. Paper descriptions suffer from perceptual limitations and lack of interactivity, and mostly important they do not provide monitoring and logging capabilities that are crucial for determining patient adherence to the program and the effectiveness of the exercises.

Fig. 1.

Example of a typical paper description of an exercise given to patients.

Our approach addresses these challenges in an unified way. We first design motion demonstration methodologies that allow therapists to intuitively create, edit and re-use customized exercises that are responsive to the needs of their patients. In this way we integrate in our systems the ability to configure exercises to particular patients, both in terms of creating new exercises as needed and in terms of designing how exercises should adapt to patient preferences, physical limitations, and recovery rates. Several factors can be also considered to adjust a system to the user’s preferences: from the language and pace to display messages and instructions, to the appearance of the virtual character demonstrating the exercises, etc. Multiple interaction channels can be customized in order to approach a similar set of communication channels that the patient is used to experience during his or her daily human-human interactions. Cultural background, social and age groups all play important roles in the wide variation of preferences that can be identified and modeled in VR systems.

We describe in this paper our first steps towards such an adaptive and responsive interactive therapy system. We discuss adaptive exercise models, motion processing algorithms, and exercise delivery and monitoring techniques that are able to effectively respond to physical limitations and recovery rates of individual patients. The presented solutions provide a basic framework to experiment and address a first set of adaptation and customization features, and we focus on adaptation of exercises for the shoulder complex. We also present remote networked solutions for allowing therapists and patients to share motion performances in real-time. The transmitted data is lightweight and remote collaboration can well scale to several patients at the same time. The capability of remote sessions is important in order to keep patients motivated and engaged in the therapy when they are supposed to work on their therapy programs at home. Remote sessions also have great potential to reduce costs and to widen health care delivery.

A number of additional features are also presented for achieving a complete framework for therapy modeling, delivery and analysis. Our system provides 3D assessment tools for monitoring range of motion, and for allowing the visualization of a number of therapy parameters during or after execution of exercises. We have implemented our system in two configurations: a low-cost version based on a Kinect sensor and a high-end version based on a full-scale immersive Powerwall (see Fig. 2).

We have collected informal feedback from therapists demonstrating that adaptive and responsive exercise delivery improves their willingness to adopt the proposed solutions in their practice.

Fig. 2.

Our VR-based collaborative system can run in two configurations: a high-end immersive setup provides improved motion capture and visualization results (left), while the Kinect-based setup provides a low-cost solution suitable for patients and therapists using traditional desktop computers (right). The overlapped virtual characters represent the user’s avatar and the autonomous character demonstrating exercises, or the user’s avatar and the avatar of the remote participant.

2 Related Work

Over the last decade serious games for rehabilitation have become an important research focus with relevant evidence of benefits [15, 26]. Different types of applications have been developed targeting both specific and broad types of applications [12, 21, 41]. Virtual reality has been successfully applied for rehabilitation of stroke patients [5, 6], and with a different purpose, fitness applications have also emerged from videogame interfaces [28] and other custom-made light devices [8].

Perry et al. [42] described the typical workflow of applications with respect to neuro-rehabilitation. The workflow in clinics follows a cyclic process of treatment planning (generation), execution (delivery) and performance assessment. The traditional physical therapy protocol follows a similar pattern and the same concept can be extended to develop applications for physical therapy.

Standard commercial physical therapy packages adopted by clinicians rely on regular media to deliver exercises. The information is usually conveyed through simple text information, sequence of images, and/or video recordings. Users are only controlled and assessed while they interact directly with physicians during in-clinic follow-ups. Patients are therefore many times left unsupervised. The use of new technologies to overcome the limitations of standard approaches to physiotherapy is becoming increasingly popular. For example, the prototype product Reflexion Vera [44] tracks and monitors users through a lightweight sensor (Microsoft Kinect or similar) reporting to the therapist each performance.

Due to the high potential, research relying on the Kinect sensor is being performed to estimate the precision and validity of the device for posture assessment [9] or for motion analysis [3, 10]. Based on these studies, Kinect can be used to reliably track some types of motions, in particular upper-body exercises [31]. Exoskeletons, robotic arms with force feedback and more precise, marker based, tracking systems have also been employed for assisting and monitoring impaired patients; however, involving cumbersome and costly devices is not suitable for widespread use [18, 43, 48].

In our approach the creation and delivery of a physical therapy program follows a programming by direct demonstration strategy. The key benefit is to allow users to intuitively define new exercises as needed. The overall approach has been adopted in many areas [7, 32, 49], and it involves the need to automatically process captured motions according to the goals of the system.

Velloso et al. [50] propose a system that extracts a movement model from a demonstrated motion to then provide high-level feedback during delivery, but without motion adaptation to user performances. The YouMove system [1] trains the user through a series of stages while providing guidance and feedback; however, also without incorporating motion adaptation to user performances. Our approach incorporates motion adaptation in several ways, allowing greater flexibility to achieve effective exercises to patients of different learning abilities, impairments, and recovery rates.

A typical approach for delivering physical therapy exercises is to track user movements while a virtual character displays the exercises to be executed. The representations of both the user and the virtual trainer are usually displayed side by side or superimposed to display motion differences, improving the learning process and the understanding of the movements [19, 51].

Automated systems often allow parameterization capabilities. For instance, Lange et al. [27] describe core elements that a VR-based intervention should address, indicating that clinicians and therapists have critical roles to play and VR systems are tools that must reflect their decisions in terms of taking into account a person’s ability to interact with a system, types of tasks, rates of progression, etc. [17, 29]. Geurts et al. [12] describe 5 mini-games that can be calibrated and adapted in terms of speed and accuracy. The physical exercises are static and cannot be replaced. In comparison, our approach is much more comprehensive in that it relies on motion capture and on processing entire full-body motions for adaptation. By doing so we propose new motion processing approaches to achieve adaptive motions that are both controllable and realistic.

Significant research on motion capture processing has been performed in the computer animation field. Motion blending techniques with motion capture data [7, 23, 33, 35, 45, 46] are popular and provide powerful interpolation-based approaches for parameterizing motions; however, they require the definition of several motion examples in order to achieve parameterization. In contrast our proposed techniques are simple and are designed to provide parameterization of a given single exercise motion. We rely both on structural knowledge of exercises and on generic constraint detection techniques, such as detection of fixed points [30, 47] and motion processing with Principal Component Analysis (PCA) [13].

Rehabilitation based on tele-consultation between two healthcare services has been studied with different technologies. In physiotherapy, tele-treatment between healthcare and community services using video has been successfully employed in study cases with elderly with stroke [25] and knee pain [52]. Using virtual reality and serious games, Golomb et al. [14] presented a system for remote rehabilitation of hands for in-home use with distributed data sharing. Several studies have also combined live video of the patient integrated with the virtual environment to augment the patients feeling of presence in the interactive space [5, 22]. In these applications video was used to provide visual feedback. This choice however does not allow direct interaction in a virtual space. Data collection of a patient performance also becomes a difficult task when users are only captured by regular video.

One development using immersive virtual reality and 3D camera imaging reconstruction has been proposed by Kurillo et al. [24]. This hybrid system allows therapists and patients to share and interact in the same virtual space. The approach however focuses on high-quality rendering and is not suitable as a low-bandwidth solution for physical therapy. An improvement of this work [38] allows the system to additionally detect human poses and assist with balance control. Although remote collaboration has been explored in different ways, a suitable overall solution for interactive sessions has not yet been integrated for remote physical therapy sessions.

We present in this paper our combined approach to achieve exercises that can be modeled by demonstration, that are responsive to the performances of users, and that can be exchanged in real-time in low-bandwidth remote therapy sessions by limiting transmission to joint-angle data.

3 Configurations and Features

We describe in this section the main functionality and configurations that we have developed in our system.

Therapists can design exercises and therapy programs, and then use the system to deliver the exercises in different ways. Created exercises and programs can be stored for further reuse and sharing. When virtual characters autonomously deliver exercises, a number of parameters describing adaptation strategies can be customized, and monitoring and logging tools can be enabled as needed. The provided tools improve patient understanding, motivation and compliance, and also provide data gathering.

Two configurations have been developed, and while the user interface is different the functionality remains the same. Both configurations can work offline, where the patient can only interact with an autonomous virtual (animated) therapist, or online, where remote patients and therapists are tracked simultaneously and their avatars are displayed in the same virtual space. In all cases a number of analysis tools for real-time or post-analysis monitoring, feedback and logging are always available.

The software application has been developed based on the Ogre3D graphics rendering engine [40]. This choice has allowed us to produce and customize a same application across different modalities and platforms. The system can be easily ported to different operating systems or to more complex virtual reality settings like CAVEs.

3.1 Immersive VR Configuration

Our experimental immersive setup consists of a Powerwall system composed of six rendering computers, a main rendering node and an external computer driving input devices and the motion capture system. The interaction with the application is fully immersive; thanks to virtual pointers and a 3D graphical user interface controlled by a Wiimote. See Figs. 2-left and 3.

Fig. 3.

Example of using the immersive Virtual Reality configuration.

The high-end configuration allows therapists to immersively model exercises by demonstration and to experience full-scale visualization of patient performances. The patient’s motion can be captured and displayed in real-time or it can be loaded from previously logged sessions. The application provides stereo visualization for enhanced comprehension of the motions and data.

A high-end system configuration also allows the integration of precise tracking capabilities. In our setup the user’s upper body motions are tracked using a 10-camera Vicon motion tracking system. For improved usability, our experimental setup is configured to only track markers attached to the hands, torso and head. The motion is calibrated and mapped to the avatar following simple scaling and reconstruction procedures, as described in the work of Camporesi et al. [7]. This solution has been enough to allow us to experiment with the system; however, since we reconstruct the motion from a reduced marker set not all degrees of freedom of the user’s motion can be precisely replicated; in particular, the elbow orbit motion around the shoulder-wrist axis is set to be always in a low-energy position. If precise motion replication is needed, in particular for cases where avoiding compensatory movements is important, additional markers have to be placed on the user.

In remote connection mode the immersive system allows to achieve full-scale interactions that are closer to how humans interact to each other. When connected to a remote site, two avatars are displayed for representing the connected patient and therapist. Previously recorded sessions can also be played on any of the avatars. The avatars can be visualized side-by-side or superimposed with transparency.

3.2 Low-Cost Configuration

The low-cost configuration is designed to be of simple installation and maintenance at clinics or at home. The patient is tracked through a markerless motion tracking device, in our case using a Microsoft Kinect sensor or similar. Such configuration is important because it is simple, portable and suitable for any kind of desktop environment. It is also suitable to assist patients in their daily routines in clinical environments. See Figs. 2-right and 7 for examples.

The Kinect imposes some limitations, such as a limited volume of capture and the overall need to maintain a posture facing the sensor. The accuracy of Kinect drops significantly when users are not facing the camera or when body occlusion occurs, and several studies are available investigating the accuracy of Kinect [9, 11, 34, 36, 39]. Overall it still provides a good balance between precision, cost and portability.

Even though the accuracy of Kinect is limited, Kinect-based configurations can also be remotely connected to other instances of the system for collaborative sessions.

3.3 Remote Collaboration

The capability of having patients and therapists to remotely interact is important because it can save travel costs, allow more frequent monitoring, and potentially increase access to health care, in particular to remote areas. The motion of each user participating to the virtual collaboration is mapped directly to each respective avatar, and the avatars can be superimposed with transparency or appear side-by-side in the applications. See Fig. 4 for examples.

Fig. 4.

Examples of collaborative sessions. Left: one user is being tracked by the high-end system Vicon cameras while the other is being tracked by a Kinect sensor. Right: both users are tracked by Kinect cameras and collaborate with the portable versions of the system running in a desktop and laptop.

The communication between two peers in a collaborative session is based on a client-server UDP communication schema with added packet ordering, guaranteed communication reliability and optional data compression. The server application, after accepting and validating an incoming connection, starts sending information of the avatar of the current user (sender) and waits the update of the client’s avatar (receiver). For instance, if the therapist application is started as a server, the therapist’s avatar becomes the active character in the communication and the second character, the patient’s avatar, becomes a receiving entity. If the patient’s application is started as the client, the sender entity becomes the character of the patient’s application while the tutor/therapist becomes a receiving entity waiting for further updates.

During a networked session each active character maintains a history containing its previous poses and the streamed information between the peers is limited to the information that has changed between the previous frame and the current frame. This feature has been developed to handle communication between peers with limited bandwidth capabilities.

Feedback and analysis tools (described below) are also available during virtual collaboration. The therapist can demonstrate exercises, analyze the patient motion, load preset exercises from the database, watch the patient’s performances, record a patient motion in real time, etc.

3.4 Tools for Real-Time Feedback and Post-analysis

The feedback tools can be activated anytime and they are highly customizable. For example, any joint of the character representation can be tracked and considered for analysis by any tool. Simple commands or text-based configuration files are used for customization. Four types of feedback have been developed in order to provide visual and quantitative information about the user motions in real-time or in post-analysis. The four feedback tools provide information with respect to: trajectories, joint angles, distance to target exercises, and range of motion per exercise. See Fig. 5.

Fig. 5.

Visualization helpers are available for real-time feedback or post-analysis of motions.

Trajectories: Trajectory trails of selected joints can be displayed in real-time, showing the performed trajectory of a joint during a fixed past period of time (see Fig. 5(a)), or after a user’s performance, showing the performed trajectory and the trajectory compliance range with the reference exercise. The visualization can be based on fine polygonal segments sampled per frame (for precise analysis for example of tremors), or smoothly generated by B-Spline interpolation.

Angle Estimation (Virtual Goniometer): Joint angles can be visualized (Fig. 5(b)) with a floating label showing the angle value and the local lines used to measure the angle. In practical goniometry for upper-limb physiotherapy [37] angle measurement is important in order to measure progress and intervention effectiveness, via therapy or via surgery. Therapists can therefore instruct specific movements to patients and observe or log the achieved measurements. The provided angle measurements match the angles measured in practical physiotherapy protocols [37]. The proposed angle measurement is simple and yet flexible to accommodate generic needs.

The angle estimation is calculated as follows: let $p_1, \cdots , p_4 \in \mathbb {R}^3$ be the global positions of the extremities of two dependent (bones sharing a joint) or independent bones, and $R_1, R_2 \in \mathbb {SO}^3$ be the user-defined reference frame rotations. The angle estimation between the limbs at the joint in question is obtained with:

$$\begin{aligned} \phi = \arccos \left( \left( R_1 * \Vert p_2 - p_1\Vert \right) \cdot \left( R_2 * \Vert p_4 - p_3\Vert \right) \right) . \end{aligned}$$

(1)

The proposed method allows the system to measure any kind of angle by just defining pairs of joints and optional reference frame rotations. The tracked angles are specified in the application’s configuration file. It gives to the therapist a flexible and easy mechanism to identify and customize the visualization. To isolate angles for upper-arm flexion (extension or abduction) we track, for instance, the angle generated by the scapula/clavicle and humerus, given the scapula bone aligned to the torso as a consequence of the skeleton hierarchical structure. The measured angle is the angle between the arm and the “body line” of the user. In default behavior, angles are only displayed when significant motion is detected. With respect to the effectiveness of using Kinect for upper-limb joint angle estimation, the approach has been tested and validated in a similar context [36].

Distances: Colored 3D arrows showing the distance between corresponding pairs of joints, each belonging to a different character, are useful for the patient to track compliance with the demonstrated exercises (see Fig. 5(b)). The feedback is useful in individual sessions or in remote physical therapy sessions. The distance arrows are employed similarly to the technique proposed by Anderson et al. [1]. The arrows are programmed to automatically disappear if the corresponding distance is under a given threshold, and different colors can be associated to different ranges of thresholds. This is in particular useful for slow exercises where compliance is important. Figure 6 shows arrow distances enabled together with several angle measurements during execution of one exercise.

Fig. 6.

Example of several feedback tools enabled while a patient executes a given exercise.

Range of Motion: Our range of motion visualization (see Fig. 5(c)) analyzes the rotation of a selected joint overtime. We focus here on the shoulder range of motion evaluation due its importance in rehabilitation of shoulder movements.

The 3 degrees of freedom (DOFs) of the shoulder joint orientation are decomposed into the twist and swing rotation parameterization [16]. The swing motion is then tracked at every frame $i$, and for each swing orientation $s_i$ measured, the intersection point $p_i$ of the upper-arm skeleton segment at orientation $s_i$ and a sphere centered at the shoulder joint is computed. The history of all traversed $p_i$ points is visualized with colors in the sphere. The sphere is texture-mapped with an image texture initially fully transparent. For every measured point $p_i$, its position in the texture is determined and the corresponding texture pixel $c_i$ has its color changed. For achieving a clear visualization we employ a relatively high texture resolution and we weight the color increments around $c_i$ with a local Gaussian distribution centered at $c_i$. The colors are incremented from pure blue to red, providing a colored frequency map of all traversed swing orientations (see Fig. 5(c)).

The boundary of the colored map will represent the range of motion executed in a given exercise. The original points $p_i$ are also recorded and are used for geometrically estimating the polygonal boundary describing the full range of motion during a session. This tool provides an excellent way to log improvement of range of motion during rehabilitation, to observe the patient’s ability to execute precise trajectories, and to observe if there are areas that are avoided for instance due pain or discomfort. In summary the representation provides a frequency history of the space traversed by the user, and it offers a comprehensive view of the patient’s performance. Frequency maps collected per exercise can clearly represent patient progress across therapy sessions.

4 Adaptive Exercises

The option of providing customized exercises by demonstration enables the therapist to go beyond recovery plans limited to a set of predefined exercises. The therapist can record his or her demonstrations and then trim, save, load, play, and customize them in different ways. After a validation process the motions can be corrected and/or parameterized. Exercises can then be saved and categorized in a database of exercises. The database is used for fast construction of therapy programs using a desktop-mode interface of the application during consultation with patients.

In order to achieve adaptive exercises we need to address exercise parameterization from the beginning, since the modeling of the exercise motion. Our approach of modeling exercises from demonstration (see Fig. 7) allows exercises to be generic; however, some structure is expected in order for motion processing algorithms to be able to parameterize the motions in real-time.

Fig. 7.

Illustration of a modeling session by demonstration using the low-cost Kinect configuration.

Given a captured exercise, we propose correction and parameterization techniques that allow (1) detection and fine-tuning of key characteristics of the exercise such as alignments and constraints, (2) parameterization of the exercise by detecting modifiable properties such as speed, wait times and amplitudes, and (3) real-time motion adaptation by monitoring user performances and updating the exercise parameters in order to improve therapy delivery.

The presented techniques facilitate the process of defining exercises by demonstration by providing several modeling and correction mechanisms and at the same time providing parameterization for real-time adaptation. As a result the proposed methods produce realistic continuous motions that can adapt to user responses in order to improve motivation and outcomes.

4.1 Detection of Geometrical Constraints

A constraint detection mechanism is designed for three specific purposes: to inform motion parameterization, to help correcting artifacts and noise in the motions, and to provide metrics for quantifying motion compliance. The metrics are used to provide visual feedback to the user informing the correctness of motion reproduction, to make decisions during the real-time adaptation mechanism, and to achieve an overall user performance score for each session.

Appropriate constraints are not constraints which are to be absolutely followed. Recorded motions may have unintended movements and imperfections introduced by the capture system. Constraints must be detected despite these fluctuations, and should be softly enforced.

We analyze the position in space of a specific joint with respect to a frame of reference $F$ which can be placed at any ancestor joint in the skeleton structure. Since different types of constraints can be recognized with respect to a specific joint (during the overall duration of the motion) a chain of constraints can be also detected and reported to the user during the parameterization process.

The detected constraints are provided to the user and the user then decides (1) if the motion should be modified to better enforce the detected constraint, and (2) if the constraint is to be monitored during real-time execution of the exercise in order to alert the user every time the constraint is significantly violated. For instance, if the elbow joint is detected to be immovable in an exercise, the system will detect that as a point constraint and may alert the user in real-time everytime the user’s elbow is too far away from its point constraint.

The detection framework can accommodate any desired type of geometric constraints, such as points, lines, circular trajectories, etc. While we focus here on describing first results with point constraints, plane constraints can be well detected by PCA analysis and the same principles are applicable to several other types of constraints.

A point constraint describes a child joint that is static relative to its parent. Let’s $P_i, i \in \{l, \ldots , k\}$ be the cloud of points formed by a joint trajectory with respect to a local frame $F$ generated by re-sampling linearly the motion frames with constant frame rate. The standard deviation of the cloud of points $\sigma$ is calculated and subsequently checked against a specific threshold $\alpha$. When the condition is met the current joint is marked as a point constraint and it is represented by the specific point located at the mean $\mu$. When a point constraint is detected the ancestor(s) can be then adjusted to enforce the constraint as an exercise correction filter for customizing the appearance ad correctness of the exercise. It is also useful to not completely correct constraints in order to keep the original humanlike appearance of the recorded motions.

The user is offered a correction percentage to choose. 100 % correction results in motion which always tries to obey constraints, whereas 0 % correction results in no modification the original motion. In a given frame, a point constraint is enforced through spherical linear interpolation between each joint orientation and the computed mean. Figure 8 illustrates results obtained.

Fig. 8.

Point constraint detection example. From left to right: the elbow motion trajectory can be gradually corrected to its mean position while the wrist motion trajectory is gradually corrected to a plane. The elbow and wrist trajectories are shown with different correction factors: 0 %, 50 %, and 95 %. Partial corrections allow to improve alignments while preserving the original naturalness of the exercise motion.

4.2 Detection of Exercise Parameterization

Consider a typical shoulder flexion exercise where the arm is raised until it reaches the vertical position or more (initial phase); subsequently the arm is hold for a few seconds (hold phase) and then it relaxes back to a rest position (return phase). This is the type of exercise that we seek to parameterize.

The analysis procedure makes the following assumptions: (a) each motion represents one cycle of a cyclic arm exercise that can be repeated an arbitrary number of times; (b) the first frame of a motion contains a posture that is in a comfortable position representing the starting point of the exercise; (c) the exercise will have two clear distinct phases: the initial phase is when the arm moves from the initial posture towards a posture of maximum exercise amplitude, then the exercise may or not have a hold phase but at some point the exercise must enter the return phase, where the exercise returns to the starting posture at the end of the exercise. This implies that the initial posture is approximately the same as the final one.

The analysis if the exercise can be parameterized starts by detecting the points of maximum amplitude in the motion in order to segment the demonstrated motion. If a mostly static period is detect near the maximum amplitude point, then that period is extracted as the hold phase. If the phase segmentation is successful the input motion is segmented in initial, return and (optionally) hold phases, and the motion can be parameterized.

If the motion can be parameterized it is then prepared for on-line parameterization. We parameterize amplitude in terms of a percentage of the wrist trajectory: 100 % means that the full amplitude observed in the input motion is to be preserved, if 80 % is given then the produced parameterized motion should go into hold or return phase when 80 % of the original amplitude is reached, and so on. Let $h$ be the time duration in seconds of the desired hold duration. When the target amplitude is reached, the posture at the target amplitude is maintained for the given duration $h$ of the desired hold phase. When the hold phase ends, the posture is then blended into the return motion at the current amplitude point towards the final frame. The blending operations ensure that a smooth motion is always produced. Velocity profile adjustment and an idle behavior are also added in order to ensure a realistic final result. See Fig. 9 for an example.

Fig. 9.

The red and blue trajectories show the initial and return phases segmented out of the input motion. (a) The full (100 %) amplitude of the input motion is shown by the trajectories. Two crosses at the end of the trajectories (in almost identical positions) mark the positions of the maximum amplitude points. (b) The two crosses now mark the maximum amplitude points in the initial and return trajectories at 75 % amplitude. (c,d) In this frontal view it is possible to notice that the postures at 75 % amplitude in the initial and return phases are slightly different. The hold phase will start by holding the posture shown in (c) while a breathing behavior is executed, and when the hold phase is over, the posture is blended into the return motion starting at the posture shown in (d), in order to produce a smooth transition into the return phase (Color figure online).

The described procedures allow us to parameterize an input motion with respect to up to four parameters: amplitude $a$ (in percentage), hold time $h$ (in seconds), wait time $w$ (in seconds), and speed $s$ (as a multiplier to the original time parameterization). Given a set of parameters $(a,h,w,s)$, the input motion can be prepared for parameterized blending operations very efficiently and then, during execution of the parameterized motion, only trivial blending operations are performed in real-time.

4.3 Real-Time Adaptation

When the adaptation mechanism is enabled the system collects information about the patients performance in real-time in order to adapt the current exercise in its next repetition. Four types of adaptation mechanisms are provided:

• Amplitude Adaptation: The range can vary from 75 % to 100 % of the target amplitude parameter. The system tracks the distance between the user’s end-effector and the point at the target amplitude position. If the minimum distance is larger than the amplitude compliance parameter specified by the therapist, the next exercise execution will have the target amplitude lowered to the position that makes the position reached by the user to become within the compliance range. If in a subsequent repetition the user reaches the current (reduced) target amplitude, then the next target amplitude will be increased towards the original target amplitude.

• Hold time: The hold phase adaptation is designed to adapt the time at hold stance to improve resistance, usually in a posture that becomes difficult to maintain over time. The maximum distance between the target hold point and the performed end-effector position is tracked. If above a threshold, the patient is having difficulty in maintaining the demonstrated posture during the hold phase and the next exercise repetition will have a shorter hold phase duration time. If in a subsequent repetition the patient is able to well maintain the hold posture, then the hold duration is increased back towards the target value.

• Speed execution: During patient monitoring, the active position of the patient’s end-effector is tracked and its distance to the demonstrated exercise end-effector is computed for every frame. If the average distance computed across the entire exercise is above a given posture compliance threshold, the next exercise execution speed is decreased. If in a subsequent repetition the difference is under the threshold the play speed will be adjusted back to the previous execution speed.

• Wait-time between exercises: this adaptation mechanism allows the system to update the waiting time between exercise repetitions. If the user is well performing the exercises a shorter wait time is allowed, otherwise a longer wait time is preferred. A target wait time is first specified in the therapy program and then it is decreased or increased according to a performance metric that is used to determine how well the patient is following the exercises overall. The metric can be customized by combining the compliance metrics used for the exercise compliance, speed compliance, and hold phase completion.

The described adaptation mechanisms have been identified as a first set of relevant strategies after many discussions and interactions with therapists. In the next section we present a summary of the main feedback received.

5 Feedback Results and Discussion

Since the beginning of the development of our system we have closely worked with therapists in order to design the described therapy creation, delivery and adaptation functionality. With our first prototype solutions developed, we have then gathered feedback on the provided functionality. The goal was to gather first impressions and to analyze how much the proposed solutions are perceived to be useful.

We gathered feedback in two phases. In the first phase, focus groups were held and open ended questions elicited multiple responses of major factors that should be considered in exercise prescription. In the second phase we demonstrated the current prototype application and then asked the therapists for their feedback.

In the first phase questionnaires were distributed to 40 staff therapists asking about the importance of individualized interactions, the factors used to determine correctness of performed exercises, and the motivational and adaptation strategies commonly used by therapists. For each question, the therapists were asked to rank the factors identified in the first phase between 1 (not important) and 5 (highest importance). The factors that were ranked 4 or 5 by more than 20 therapists are summarized in Fig. 10 for selected questions.

Fig. 10.

Summarized responses. The numbers show how many therapists (out of 40) rated each factor as 4 or 5.

From the collected data summarized in Fig. 10 it is possible to make several observations. Performing exercises in a correct manner is largely related to being close to the prescribed exercises, what is translated in terms of not having compensatory movements and maintaining correct postures and trajectories. The several visual feedback tools that were described well address these issues. In addition, the proposed constraint detection methods for real-time warning if the user performs motions that do not well respect constraints also well address enforcing correct execution of exercises.

One point that cannot be addressed by therapy systems that only give visual output is to provide tactile feedback. However we point out that tactile feedback was considered as important as visual feedback, which is well addressed by our system. The fact that visual and audio feedback were highly ranked is also important because it indicates that they may well compensate for the lack of tactile feedback, which is at the same time a desirable characteristic of the system from a safety perspective.

Several causes were cited as reasons justifying the need for exercise adaptation, for example, the patient’s ability to learn, patient improvement, decreased pain, increased strength, etc. The proposed adjustment of wait and hold times, exercise speed and amplitude provide direct ways to adapt the exercise as the patient progresses. In particular, it is also important to adapt in a constant basis given the patient’s ability to learn the exercises. At the beginning of an exercise set it is often observed that patients need more time to assimilate an exercise while at the end of a set the patients are well able to perform them quicker and with less wait times. The same can be observed in subsequent sessions, however usually with progressively faster learning rates. The proposed adaptation methods are capable to adjust to patients as needed, thus significantly improving correct exercise execution, improvement observation, engagement, and adherence to the therapy program.

In the second phase we have asked questions to the participants before and after they have seen the capabilities of our system. The questionnaire consisted of generic questions as a follow up of the first phase questionnaire, and it also included open-ended suggestions and preferences to improve the current setup. At first only 45 % of the participants were confident that patients do exercises consistently and correctly at home, but after seeing our system (and if our system was to be employed) that percentage raised to 70 %. When asked about the importance of modifying exercises during the progress of the therapy, 70 % cited as very important and after seeing our system this percentage was even raised to 85 %.

These results indicate that adaptation is an important factor for achieving VR-based therapy systems that have real potential to be adopted. While the presented solutions provide only first results towards addressing adaptation strategies, we believe that the described framework and adaptation techniques provide a significant step towards the right direction. Many variations and adjustments to the described procedures are possible, and our solutions are being fine-tuned in preparation for validation activities with patients.

6 Conclusions

We present innovative solutions based on VR technologies for addressing limitations of traditional upper-limb physical therapy. The proposed solutions simplify the work of therapists and also help patients during their daily exercise routines with adaptive exercise delivery strategies that improve engagement and adherence to the therapy program. As future work, our solutions are being fine-tuned in preparation for validation activities in real practice.