Section 1: Motion Sensing in Healthcare

There is an urgent need in healthcare for the development of functional, accurate, affordable, and scalable systems that can provide physicians with actionable information in order to advance healthcare delivery. Motion monitoring platforms meet this need by providing physicians and researchers with the tools to effectively measure the type, quantity, and quality of patient activity in order to improve care and establish cost-effective, evidence-based practices. Furthermore, the small form factor and low power consumption of microelectromechanical system (MEMS) based motion monitoring platforms enable the development of novel systems that can provide remote point-of-care diagnostics and continuous long term monitoring.

In neurological rehabilitation, for example, motion monitoring can provide solutions for frequent problems faced by physicians including: measuring the gains and losses of daily function over time, assessing compliance of prescribed exercise, and providing more frequent performance feedback, enabling physicians to more quickly update patient instructions [1]. Additionally, portable motion monitoring platforms provide remote access to laboratory-quality data, enabling the evaluation of conditions difficult to observe clinically and provide an ecological alternative to expensive and time consuming laboratory evaluations [2].

Popular consumer motion trackers (e.g. Fuelband, FitBit, MisFit) capable of providing basic physiological information and activity classification for healthy patients have proven to be unreliable in accurate characterization of subject motion [3, 4]. These devices, typically mounted on the wrist, utilize low power triaxial accelerometers to detect episodic movements which are assessed in real time for patterns of acceleration and deceleration. Adventitious movements that match internal algorithms may be interpreted as a motion of interest while abnormal or weak movements that don’t meet the necessary thresholds may be ignored [1]. Inaccuracies are further exasperated when used by individuals with physical disabilities that exhibit slow or abnormal movements [5]. Additionally, the classification algorithms employed by fitness trackers suffer from either a small activity set or low accuracy which limit the range of useful applications [6]. Algorithms such as those employed by [7, 8] decline in accuracy as the number of potential motions increases and very few are able to produce meaningful metrics as the classifiers were designed without consideration for the fine biomechanics of motion. Thus, in their present configuration motion trackers are not suitable for use in healthcare.

To meet the demands of healthcare, motion monitoring platforms must combine a multitude of sensors with clinically proven machine-learning algorithms that enable large-scale networked systems with automated activity profiling to provide physicians with accurate, reliable, and relevant information. Clinical trials utilizing purpose built motion monitoring platforms have shown to accurately detect the presence and severity of various diseases, including Alzheimer’s [9], Parkinson’s [10], and sleep apnea [11]. In addition to diagnosis, these motion sensors have enabled researchers to monitor disease progression and therapy effectiveness [12].

Section 2: Motion Sensing Devices

Visual and inertial sensors platforms are the two most popular technologies used for human motion sensing. In this section, we provide a brief introduction to the two systems as well as comparing their capabilities and limitations. Additionally, we discuss the great advances provided by combining the two sensing technologies resulting in a system with more reliable motion inference. Furthermore, examples of sensor fusing algorithms are presented that address errors due to sensor measurement and sensor placement.

Vision-Based System

Vision-based motion sensing systems comprise of two major categories: marker-based systems and image-based systems.

Marker-based motion capture systems [13, 14] track the movement of reflective markers or light-emitting diodes placed on the human body, thus indirectly track the movement of body segments as well as the configuration of body joints. For such systems, accurate 3D marker positions in a global frame of reference are computed from the images captured by a group of surrounding cameras using triangulation. Although such systems can provide high-precision joint position in 3D space, they are extremely expensive and time intensive in their deployment. Therefore, they are infeasible for daily activity monitoring.

Marker-less systems use computer vision techniques to derive motion parameters from the captured video [15]. Recently, low-cost off-the-shelf sensors have exploit depth cameras to capture the movement of human limbs and extract the 3D position of body joints. The Kinect, for example, is a motion-tracking device developed by Microsoft capable of monitoring up to six full skeletons within the sensors field of view. For each skeleton, 24 joints are defined and their positions and rotations tracked. Due to the embedded tracking algorithm’s large training data set, the Kinect provides accurate tracking outcomes which can be considered as the ground truth [16]. Another example is the Leap Motion controller, which is designed specifically for motion tracking of hand gestures. In this system, three infrared LEDs and two monochromatic cameras are used to reconstruct the 3D scene and precisely track hand position within a small range. Research suggests that the Leap Motion controller can potential be extended as a rehabilitation tool in the home environment, removing the requirement for the presence of a therapist [17].

While vision-based systems can provide desirable tracking accuracy, they are not self-contained and require cameras deployed in the environment. Additionally, vision based systems raise privacy concerns and are as yet not feasible for large-scale employment.

Inertial Sensor Based System

Advances in MEMS technologies have led to the proliferation of wearable inertial sensor based activity monitoring systems. State-of-the-art inertial sensing platforms typically include: accelerometers and gyroscopes. MEMS accelerometers sense the sum of accelerations contributed by gravitation acceleration and motion of the sensor relative to an inertial reference frame. Detection of acceleration is determined by measuring the change in capacitance resulting from displacement between silicon microstructures forming capacitive plates. The measured capacitance may then be applied to compute acceleration. The MEMS gyroscope measures the Coriolis force exerted by a vibrating silicon micro-machine mass on its flexible silicon supports when the sensor undergoes rotation. Silicon microstructures within the gyroscope use electrostatic forces exerted through capacitive plates to vibrate the suspended proof mass. The Coriolis force, often referred to as a fictitious force, represents a mass acting on an object moving in a rotating reference frame. Rotation of the sensor induces the Coriolis force leading to a displacement of the proof mass that is proportional to the angular rotational rate. A diagram showing a typical MEMS accelerometer and gyroscope architecture are shown in Fig. 1.

Fig. 1
figure 1

Typical MEMS architecture diagram showing (a) single axis accelerometer sensitive to acceleration in the direction of the indicated arrows and (b) single axis gyroscope sensitive to the rate of rotation for a rotation vector perpendicular to the page

Full size image

Activity monitoring using MEMS inertial sensors is rapidly growing. Reference [18] used one triaxial accelerometer mounted on the waist to classify activities correlated with movements measured in a controlled laboratory. References [19, 20] utilize a Kalman filter to combine accelerometer, gyroscope, and magnetometer sensor data to detect slow moving body rotation and linear translation. In [21], the author developed a biomechanical model to track motions with wearable sensors.

Furthermore, inertial sensor based activity monitoring systems have been verified to accurately and reliably characterize the gait of post-stroke patients [22, 23]. In a large scale clinical trial, a group of physicians and engineers deployed wearable inertial devices on hundreds of post-stroke patients with feedback provided to the physicians and patients on a daily basis. The system proved effective in monitoring activity in the ambulatory community [24, 25].

To detect relative position in 3D space, data from inertial sensors require double integration. Thus, the drift and broadband noise present in MEMS sensor result in rapid accumulation of errors. To meet the stringent accuracy requirements for use in healthcare, algorithms must be developed to reduce the impact of noise on the final results.

Sensor Fusion of Optical and Inertial Sensing Technologies

With the capabilities and limitations of the above two sensing technologies, sensor fusion algorithms can be applied to infer subject motion state.

Reference [26] proposed the use of the Kinect system to determine calibration errors of inertial sensors. The author used a Kalman filter to integrate the Kinect data with noisy inertial measurements to improve the overall tracking outcomes. Satisfactory results were obtained through experimentation on healthy subjects performing various tasks.

Reference [27] demonstrated a system shown in Fig. 2 which fused the Kinect and inertial sensors to achieve opportunistic calibration of sensor placement errors. Position data obtained from the Kinect were first smoothed and converted to virtual measurements (virtual accelerations), which served as the ground truth. The system opportunistically used this ground truth to detect and compensate placement errors of inertial sensors. Experiment results indicated that the system could accurately reconstruct motion trajectories of upper limbs among healthy subjects even when the sensors were misplaced.

Fig. 2
figure 2

(a) Subject standing in front of the Kinect sensor with inertial sensors placed on the wrist. (b) Virtual reconstruction of the subject by the Kinect sensor. Data from both the Kinect and inertial sensors are fused to achieve opportunistic calibration of sensor placement errors

Full size image

Section 3: Motion Data Processing

A system supported by multiple inertial sensors with ideal measurement characteristics may enable computation of accurate subject body motion based upon direct kinematic computation. However, MEMS gyroscope and accelerometer systems present errors due to ill characterized drift in measurement which accumulate rapidly with subsequent integration appearing in kinematic computation [28]. One approach to avoid computation errors relies not upon absolute measures of acceleration and rotation, but rather, the use of classification techniques to differentiate a predefined activity set from unique features extracted from the inertial sensor data [29]. Despite its wide employment in the state-of-the-art activity monitoring systems, this method suffers from several shortcomings. First, though most activity classification systems are very successful in classifying periodic activities (e.g. lower body activities such as walking or running), their capability to differentiate upper body activities for example, eating or typing, is largely limited. Second, most activity classification systems lack the knowledge of detailed kinematic motions that are vital for healthcare. For example, metrics including gait symmetry extracted from the motion data can provide insight about the control of walking among post-stroke patients, which may have a role in guiding the clinician’s treatment decisions [30]. Third, classification performance usually degrades with larger activity sets [6]. Thus, the current activity classification systems still suffer from scalability problems.

In this chapter, a new approach is described that enables an advance in activity classification accuracy. This is based on a method relying upon subject motion context. This finally leads to a context-drive activity classification and motion tracking system. This system provides a robust activity monitoring platform consisting of three subsystems, context detection, context-driven activity classification, and activity specific motion tracking. In the following sections, algorithms for each subsystem will be described.

Context Detection

For accurate activity classification, there are two kinds of contexts that are of interest. One is physical context denoting a subset of a subject’s physiological measures such as heartbeat and body core temperature. The other is context associated with characteristic of the subject’s surround environment and the subject’s location. While the physiological context can be easily determined by using wearable devices, methods to determine a subject’s environment and location present an additional challenge.

Here we focus on location categories to describe a subject’s environmental context. This may include both location in space as well as a description of location characteristics. Of course, conventional global position systems (GPS) may indicate a subject’s location in space. However, through the use of mapping methods, such as data provided by the Google Place API, a description of a location may be obtained. For example, the city of residence, a retail environment, or a gymnasium. Determination of location environmental characteristics provides a benefit for subject motion classification. Detailed characteristics may help pre-classify some upper body activities, for example, the act of eating may occur in a restaurant location.

To determine detailed location characteristic requires knowledge of the subject’s indoor position, where GPS localization may not be available. Thus, a foot mounted inertial sensor based sensor solution, including a novel navigation algorithm has been developed. The same inertial sensors previously used for activity classification and motion tracking, shown in Fig. 3, can be utilized, requiring no additional hardware for context detection. The combination of this navigational method and indoor map data was used to infer the subjects absolute position in the environment. This method exploited also the use of a particle filter for correction of navigational drift error [31].

Fig. 3
figure 3

Inertial sensor system within a sealed enclosure

Full size image

Additionally, performance in accuracy and computational throughput can be enhanced by exploiting other sources of localization, including the discovery of WiFi access points that may exist in an indoor environment [32].

Context Driven Activity Classification

A hybrid decision tree is able to classify a large lower body activity set with high accuracy after optimizing the activity set, the feature set, and the classifier at each internal node [33]. However, experiments indicate that the algorithm performance deteriorates after including upper body activities. To enable large scale activity monitoring, context driven activity classification is introduced [34]. This framework allows personalization, which can greatly improve the classification performance. Here, personalization is enabled on two levels. First, individuals may have different sets of contexts under which activity classification is required. Furthermore, within each context, a set of individualized activities of interest may be present. This leads to the context specific activity models, resulting in increased classification accuracy, faster classification rate, and improved battery usage efficiency.

However, the above work requires additional sensors (e.g. audio sensor) to determine a subject’s context. Therefore, in [6] context is simplified to broad location categories. This simplification adversely limits the classification capability of the entire system. For example, a variety of activities can be performed in residence including eating, typing, and running. Thus, it is necessary to know the subject’s location in greater detail. By determining the subject’s environment (e.g. dining room or study), eating can be more accurately differentiated from typing.

An important advance was developed through a system utilizing inertial sensors placed on the subject’s elbows, wrists and feet to monitor their daily activities. Data from the sensors were first used to determine the user’s environment, which was separated into several location categories. This was followed by a classification algorithm [33] that utilized the location category to reduce the size of the decision tree. The classification accuracy of the subject with location information was determined to be 99% compared to the 78% accuracy obtained without location information [32]. Table 1 lists the activity sets associated with each location category used in the classification algorithm.

Table 1 Location categories narrow the possible set of activities used the classification algorithm
Full size table

Activity Specific Motion Tracking

When analyzing motions, the human body can be decomposed into nine segments [35]. One method to fully track the motion of the human body is to attach inertial sensors on each of the body segments and use a kinematic chain to model the movements [36]. However, this approach suffers from several shortcomings. First, it requires excessive computation, as both the number of state transition equations and their complexity are proportional to the number of sensors. Second, the system will be vulnerable to errors resulting from sensor misplacement. This is due to the tracking algorithms requirement to know sensor orientation in the body frame, which is usually assumed to be constant. Third, the algorithm is inefficient in distinguish specific movements that representing activity of clinical assessment value.

Therefore, in this subsection, we introduce the framework of activity specific motion tracking. Based on the results from the context driven activity classification system, the activity set can be further grouped into upper body activities (e.g. eating, typing, etc.), lower body activities (e.g. walking, running, etc.), or sports activities such as cycling. For each category, the requirements of the tracking protocol are specified. This includes the sensor set, the kinematic model, and the error reduction algorithm. In the following paragraphs, we cover the basics of the tracking protocol for each activity category.

For tracking of upper body activities inertial sensors mounted on the subject’s elbow and wrist were utilized. A complimentary filter [37], combining accelerometer and gyroscope data were used to calculate the sensor orientation and remove drift error. Through the assumption that upper limbs are rigid and no relative movements exist between the sensors and the attached limbs, orientation of the upper arm can be approximated with that of the elbow sensor. Likewise, orientation of the lower arm can be approximated with that of the wrist sensor [38]. To align the reference frames of the two sensors, a calibration method is proposed. The calibration creates a uniform reference frame allowing the reconstruction and visualization of the upper limb movements [38]. Metrics including the range of motion of the elbow joints can then be estimated by calculating the angle between the upper and lower arms.

To verify the upper body motion tracking algorithm, three female and three male subjects with varying heights performed a range of arm motions after sensor calibration. A Kinect system was used to capture the skeletal movements and record the shoulder, elbow, and wrist positions in the individual frames. Based on the rigid link assumption, the upper arm and the entire arm lengths were estimated as the distance from the shoulder to elbow and from the shoulder to wrist respectively. Table 2 presents the estimation accuracy of the calibration algorithm compared to the Kinect captured ground truth (the Kinect system can report positions to within 2–5 cm of true value). Overall, the average error was calculated to be 4.53%. In addition, the arm motion reconstructed from the inertial sensors were compared with the trajectory captured by the Kinect sensors. The results show that our algorithm was able to accurately reconstruct a variety of upper body motions.

Table 2 Algorithm estimated arm length and deviation from the Kinect sensor for subjects S1 through S6
Full size table

For lower body activities, a single foot mounted inertial sensor is sufficient. Utilizing the algorithm for upper body motion tracking the foot orientation can be calculated. This information is used to project the accelerometer data into the global reference frame, which enables gravity subtraction, leaving only the acceleration generated by the foot. Since integration will lead to large drift errors, zero velocity update (ZUPT) [39] is essential in obtaining more accurate foot velocities based on the acceleration data. A second integration can be performed to determine the position trajectories of the foot. With the calculated foot orientation and position trajectories, metrics such as walking distance, walking speed, and gait symmetry can be extracted [40] (Fig. 4).

Fig. 4
figure 4

Plot showing: (a) captured accelerometer data, (b) the double integrated result including drift, (c) estimated linear drift, and (d) double integrated result after ZUPT is used to remove drift

Full size image

To validate the lower body motion tracking algorithm, three healthy subjects were recruited. Each subject performed two sets of 40-m level walking, ten-step stair ascending, and ten-step stair descending. A sensor based reconstruction of the foot trajectory for each test is illustrated in Fig. 5. The reconstructed foot position and orientation of individual steps were compared with data captured from a Vicon video motion system. The highly accurate Vicon system is capable of measuring step length with a standard error of 0.02 cm and gait velocity with a standard error of 0.06 m/s. The results showed that the lower body tracking algorithm was able to accurately reconstruct a variety of lower body motions, achieving an absolute error of (3.08 ± 1.77)% for the total travel distance by both the left and right feet [32].

Fig. 5
figure 5

Sensor based reconstruction of foot trajectory during stair ascent, stair descent, and level walking

Full size image

For sports activities, additional motion tracking protocols may be required. Described here is the protocol to track lower body motions during cycling. Similar to lower body activities, a single foot mounted inertial sensor is used to calculate foot orientation during a cycle stroke. However, unlike walking or running, cycling does not contain any stationary phases of the foot, requiring an alternative to ZUPT for reducing sensor drift. A unique characteristic of cycling is the repetitive circular motion of the feet when a cyclist is pedaling. Through analysis of the accelerometer data, four waypoints along the circular trajectory (top, bottom, left, and right) can be recognized. Utilizing the four waypoints, linear interpolation can be applied to infer the foot position during the entire stroke [41]. Though the algorithm cannot predict the estimated foot position at a specific point in time, metrics such as cadence are not affected by interpolation. Similar targeted protocols may be developed for additional sports activities.

Section 4: System Implementation

In this section, motion sensor systems are discussed in more detail. First, an accelerometer only system capable of classifying daily activity and providing daily performance parameters is discussed [12, 42]. Second, activity motion tracking algorithms utilizing gyroscope measurements and both lower body ZUPT [43] and non-ZUPT [44] we will be presented.

Accelerometer Only Systems

Triaxial accelerometers are the most widely used inertial sensors due to their energy efficiency and industrial availability. The data output by accelerometers includes both gravitational acceleration as well as motion of the sensor relative to an inertial reference frame, of which both can be used for human activity recognition.

References [12, 44] present one example of an accelerometer based activity monitoring system. In the Stroke Inpatient Rehabilitation Reinforcement of ACTivity (SIRRACT) clinical trial, a sensor was placed on each of the participant’s ankles in the morning and removed at night. A Velcro strap secured each sensor proximal to the medial malleolus, flush against the bony tibia. Upon removal, each sensor was placed on a wireless power pad for recharge overnight. While charging, data stored from the sensor was automatically transferred via Bluetooth to an Android phone running a custom application. The Android phone subsequently packaged and transmitted the data via a cellular network to a secure central server for classification. The components of the SIRRACT sensor kit is shown in Fig. 6.

Fig. 6
figure 6

Components of the SIRRACT sensor kit supplied to subjects is shown. At lower left is the system smartphone. At upper center is the ankle worn Velcro attachment for the sensor. The wireless charging unit with a recess accepting the sensor is at lower center. The motion sensor system is shown at lower right

Full size image

Because gait speed as well as stand and swing symmetry varies greatly among the post stroke rehabilitation patient population, templates were generated for each participant’s gait from a set of standardized walks. Prior to receiving a sensor kit, each participant was asked to perform stopwatch-timed 30-ft walks at self-selected slow, normal, and fast speeds. These walking bouts were applied as templates in training of a Naïve Bayes classifier algorithm. Every 2 weeks, additional templates were collected to refine the model parameters and measure the changes in the patient’s gait.

After each participants’ daily motion data were uploaded onto the server, the binary classifier automatically labelled the walking segments. Subsequently, gait parameters such as walking speed and walking duration for each identified walking bout was calculated and compiled into a profile quantitatively describing the gait performance. A full list of all the metrics classified by the SIRRACT system can be found in Table 3. In addition, summaries of the metrics were made available to the therapists.

Table 3 List of metrics reported by the SIRRACT clinical trial
Full size table

Accelerometer and Gyroscope Systems

Though the accelerometer system provided a general understanding of post stroke activity levels, it lacked the detailed motion trajectory reconstruction that would enable physicians to better understand the rehabilitation process of a gait-impaired patient. With the inclusion of a gyroscope, the need for improved motion tracking can be fulfilled.

Reference [43] discusses a Zero Velocity Update (ZUPT) method that uses both accelerometer and gyroscope measurements to track lower body motions. Sensor orientation was calculated through the use of a complementary filter that combined both the accelerometer and gyroscope measurements. This enabled the subtraction of the gravity component from the accelerometer with the remaining acceleration due solely to motion. Double integration of the motion acceleration with zero-velocity update resulted in accurate trajectory reconstruction in three-dimensional space [44,45,46].

In order to meet the clinician’s preference for ankle-mounted lower body motion tracking sensors [1, 42, 47], the Non-Zero Velocity Update (Non-ZUPT) method was developed that allowed motion tracking systems with accuracies comparable to ZUPT [44]. This paper modifies the ZUPT method by updating the expected velocity with a non-zero value during the stance phase.

For comparison of the ZUPT and non-ZUPT algorithms, two inertial sensors were mounted on either the shoes [43] or on the ankles [44]. The sensors collected accelerometer, gyroscope, as well as quaternion orientation data at 200 Hz. Data were transmitted through the on-board Bluetooth chipset to a PC and locally time synchronized.

Both the ZUPT and non-ZUPT systems allowed for full 3-dimensional motion trajectory reconstruction with the minimal number of sensors and resulting in an average step-length estimation accuracy of 98.99% [43] and 96.42% [44] over the testing datasets.

Section 5: Summary

This chapter has presented the current state of activity monitoring for health and wellness applications. Novel activity monitoring platforms that supply data from inertial and visual sensors were discussed. Clinically proven, machine-learning algorithms enabling the classification of a wide range of activities were described. The applications resulting from motion monitoring platforms that combine the aforementioned sensors and algorithms were shown to provide physicians with actionable information to improve patient diagnosis and advance healthcare delivery. One application, utilizing a custom platform developed for neurological clinical trials was presented to show the critical benefits provided to healthcare by the new generation of wearable motion monitoring systems.