Keywords

1 Introduction

Regular physical activity is highly recommended and is known to improve both physiological and psychological health [41]. Physical exercise can be divided into two categories, aerobic activity (promoting mainly cardiovascular health) and strength training (beneficial for the whole musculoskeletal system [45], not only muscle strength). These are all good arguments to exercise [13], but it is especially important for frail populations to help them maintain functional independence [20, 41]. However, improperly executed, physical activity may cause injury [9, 16]; hence, tools to allow for exercising efficiently and at minimal risk are desirable, still no general purpose systems are yet available for purchase.

In this chapter, this lack is addressed by describing the PAMAP system. The system’s modular design allows for efficient customization, which could enable support tools as described above. Two use cases are studied in more detail: (1) monitoring of aerobic activities and (2) monitoring of strength exercises. For the former, a state-of-the-art boosting algorithm is provided. For the latter, different methodologies necessary for supervising exercises are outlined. In both cases, personalization is emphasized and the described methods are evaluated using data from field trials.

This chapter is intended to be interesting for both generally knowledgeable readers with a general interest for current advances in (AAL) solutions to activities of daily living ADL monitoring and support for strength activities using inertial sensing. Specialists in the field with interest in machine learning and multivariate signal pattern recognition looking for detailed algorithm descriptions to solve the aforementioned exercise monitoring problems.

This book chapter is organized as follows: Sect. 2 provides a short overview and the definitions of the most important terms that will be used throughout this chapter. Section 3 starts by presenting a generic platform concept for (AAL) systems. It describes the important components as well as the modular and flexible architecture for a physical activity monitoring using wearable sensors. Following, Sect. 3.2 showcases the implementation of the generic platform concept for the aerobic activity monitoring use case. Then, Sect. 3.3 illustrates the implementation for the strength exercise monitoring use case. Both use cases focus on the objectives, the initial requirements as well as the personalization of the (AAL) system. Section 4 briefly outlines the problems of the current setup, its implementations and the research challenges that should be addressed in future. Finally, for the general reader, a recommended reading list is presented in Sect. 4.

2 Glossary

  • ADL. The term activities of daily living summarizes daily activities within an individual’s place or in outdoor environments. The term is mainly related to health care.

  • EHR. An electronic health record is a computerized record of a person or patient. Generally, it includes the history of illnesses, diagnosis, medical treatments, etc.

  • FITT. The frequency, intensity, time, type principle can be considered as a set of rules in order to benefit from any form of fitness training and is applicable to individual exercise training.

  • IMU. An inertial measurement unit usually combines multiple accelerometers, gyroscopes, and magnetometers providing measurements of linear acceleration, angular rate, and magnetic field.

  • GPS. A global positioning system determines the position (latitude and longitude) of a receiver on earth by calculating the time difference for signals from different satellites to reach the receiver.

  • MET. The metabolic equivalent of task is a unit to measure the energy cost of physical activities. It is used to estimate the amount of oxygen used by the body during physical activity and is defined as the ratio of metabolic rate during a specific physical activity to a reference metabolic rate.

  • PSD. The power spectral density describes how the power of a signal or time series is distributed over the different frequencies.

3 A State-of-the-Art System Example

This section first proposes an AAL system architecture for physical activity monitoring using wearable sensors. It then details the two use cases, monitoring of aerobic activities and supporting strength activities, in terms of requirements, hardware platform, and monitoring methodology. The discussion is supported with experimental results.

3.1 System Overview

The proposed system is modular and flexible. As illustrated in Fig. 1, it comprises four major components that communicate with each other over network using efficient protocols. The individual components (data collection, data processing, data presentation, and data management) are outlined in the following.

Fig. 1.
figure 1

The proposed modular monitoring platform architecture.

Full size image

The data collection component is based on a network of sensors and possibly a mobile processing unit worn by the user. The sensor network can include, e.g.,  miniature inertial sensors and global positioning system (GPS) for measuring the users’ motions and/or physiological sensors, such as a heart rate monitor, ECG, skin conductivity sensor. During the data collection the data is preprocessed; the raw sensor data is corrected, filtered, and synchronized, and higher-level information, such as body pose based on multiple body-worn inertial sensors, is derived.

The data processing component analyzes and characterizes the physical activity of the user using preprocessed data. This component encapsulates algorithms developed for enabling sophisticated analysis. This can range from the derivation of the general frequency, intensity, time, type principle (FITT) parameters to the accurate evaluation of strength exercises. One of the key points in this context is to provide easy means for personalization in order to be able to target individuals or groups with specific needs.

The data presentation component provides reminders and physical activity visualization, guidance, and feedback to the user while training. Online user interfaces can range from a simplistic mobile device interface (a smartphone app), that provides alerts and just-in-time information about the current activity or daily profile. While being on the move, to a complete digital exercise trainer shown on a stationary display at home or in the gym, that guides a user through an exercise session whilst providing feedback on the way the exercises are performed.

The data management component connects the monitoring system to a private or public cloud. In a medical context, this could include uploading activity data to an electronic health record EHR enabling reviewing of this data by health care professionals. In a private context, activity data could be uploaded to social communities and shared with informal carers, such as family or friends. This sharing could promote friendly competition and motivate users to improve their performance.

By providing standards for the system components described above, the different components can easily be exchanged. This way the functionality of the system can be easily extended and specialized to create various target applications based on the same generic architecture.

The following sections showcase the implementation for the use case of aerobic activity monitoring Sect. 3.2 and strength exercise monitoring Sect. 3.3. The topics data collection and data processing will be described in detail.

3.2 Aerobic Activity Monitoring

In the field of physical activity monitoring, the recognition of basic aerobic activities (such as walking, running or cycling) and basic postures (lying, sitting, standing) is well researched and is possible with just one 3D-accelerometer [11, 18]. However, since these approaches only consider a limited set of similar activities, they only apply to specific scenarios. Therefore, current research focuses among others on increasing the number of activities to recognize, with the goal to give a more accurate and more detailed description of an individual’s daily routine [3].

Another challenge in this research field is the monitoring of physical activities in real life scenarios, which is usually neglected or even completely ignored. Moreover, recent benchmark results on physical activity monitoring datasets show that the difficulty of the more complex classification problems appearing in real life situations exceeds the potential of existing classifiers [32, 33].

This section addresses these shortcomings by describing a robust activity monitoring system for everyday life, as instantiation of the above described overall system architecture for the aerobic activity monitoring use case. The focus lies thereby on the presentation and evaluation of algorithms for monitoring a large and extensible set of activities of daily living based on a system for long-term and everyday use. Furthermore, the personalization of activity recognition algorithms — a new topic of interest in this field [19, 23] — will also be addressed, considering its feasibility in mobile applications and its applicability in everyday life situations.

Objectives and Requirements. The aerobic activity monitoring use case has two main objectives: to estimate the intensity of performed activities and to identify the aerobic activities traditionally recommended. The former objective is motivated by the goal to tell how far individuals meet physical activity recommendations, such as given in [13]. For this purpose, the system should distinguish activities of light, moderate, and vigorous effort. The ground truth for this rough intensity estimation is based on the metabolic equivalent (MET) of physical activities, provided by [1]. Moreover, to give a more detailed description of an individual’s daily routine, an activity recognition task is defined. The goal thereby is the recognition of a few (recommended) activities and postures of interest, but as part of a classification problem where a large amount of other activities are included as well. This simulates the common behavior of how activity monitoring systems are used in real life scenarios.

Due to the special characteristics of classification problems defined on aerobic activity monitoring tasks, the evaluation methodology of such systems deserves a few remarks. The commonly used standard \(k\)-fold cross-validation (CV) only simulates the scenario in which a classifier was trained. The evaluation this way is limited to the known set of users and activities, thus delivering “optimistic” results for real life scenarios. The simulation of subject independence can be achieved with leave-one-subject-out (LOSO) CV. Moreover, to simulate when unknown activities are performed, leave-one-activity-out (LOAO) CV is recommended. The combination of LOSO and LOAO evaluation gives the best simulation of how developed methods would behave in everyday life scenarios, as described in [30]. Finally, it should be noted that traditional performance measures are used in this section to quantify the classification performance: precision, recall, F-measure, and accuracy.

Data Collection. Since aerobic activities are monitored over a long period in daily life, the hardware system for this use case has important constraints to adhere: only a limited number of sensors can be used and only relaxed requirements can be defined for calibration and fixation of the sensors. Therefore, a mobile and unobtrusive system is proposed, consisting of the following components: three wireless inertial sensors (attached on the chest, over the wrist on the dominant arm and on the dominant side’s ankle, respectively), a wireless heart rate monitor, and a mobile unit for data collection, processing, and online feedback. An analysis of the necessity of the different sensors showed that this proposed sensor setup is the minimum required to achieve an accurate monitoring and assessment of the user’s aerobic activities in daily life [31].

Table 1. Definition of the intensity estimation task: mapping of physical activities included in the PAMAP2 dataset to the three intensity classes.
Full size table

Due to the lack of commonly used, standard datasets in the field of physical activity monitoring the described hardware system was used to record a new dataset [32, 33]. The created PAMAP2 dataset includes inertial and heart rate data from \(9\) subjects performing \(18\) different physical activities. The categorization of the latter into intensity classes is given in Table 1. The dataset not only includes basic activities and postures (lying, sitting/standing, walking, running, cycling and Nordic walking) traditionally used in the activity monitoring field, but also a wide range of everyday, household and fitness activities (e.g., car driving, vacuum cleaning or playing soccer). Therefore, it is suitable for defining complex classification tasks and to simulate developed methods under realistic conditions. The dataset has been made publicly available in the UCI machine learning repository [27] and will be used in the rest of this section for evaluation purposes.

Data Processing. The PAMAP2 dataset provides raw sensory data. Therefore, a common data processing chain (DPC) is applied to obtain the aimed intensity and activity class. The DPC consists of preprocessing, segmentation, feature extraction, and classification steps, as depicted in Fig. 2. The preprocessing step provides synchronized, timestamped, and labeled acceleration and heart rate data. This data is then segmented using a sliding window. Previous work shows (e.g., [15]) that for segmentation there is no single best window length for all activities. To obtain at least two or three periods of all different periodic movements, a window length of about \(3\) to \(5\) s is reasonable. Furthermore, to assure an effective discrete Fourier transform computation for the frequency domain features, a window size of \(512\) samples was selected. Since the sampling rate of the raw sensory data was \(100\) Hz in the recorded PAMAP2 dataset, the segmentation step results in signal windows of \(5.12\) s length. Thus, the preprocessed data is segmented using a sliding window with the defined \(5.12\) s of window size, shifted by \(1\) s between consecutive windows. On each of these segments, various signal features are computed in both time (e.g., mean, standard deviation) and frequency domain (e.g., energy, entropy). In total, \(137\) features are extracted, which then serve as input to the classification step. The entire DPC is described in more detail in [32].

Fig. 2.
figure 2

The data processing chain applied in the aerobic activity monitoring use case.

Full size image

To deal with the other activities in the activity recognition task different models have been proposed. Common approaches include regarding all background activities as separate activity classes (‘allSeparate’ model), introducing a single background activity class (‘bgClass’ model, basically a null-class approach) or separating the basic and background activities in a classification step before or after the actual differentiation of the activities of interest (‘preReject’ or ‘postReject’ model, respectively). With the application of the above described LOSO and LOAO evaluation techniques the model with best generalization characteristics can be identified. As shown in [30], the ‘bgClass’ model behaves the most robust in real life scenarios and will thus be used hereafter.

A wide range of classification methods has been proposed and applied in the literature of physical activity monitoring. Most common choices are supervised learning approaches such as decision trees, Bayesian or instance based classifiers, support vector machines, neural networks, etc. A comparison of different methods applied for activity recognition can be found, e.g., in [2, 24]. Moreover, some of the above listed classifiers have been used as part of an ensemble or meta-level classifier. A benchmark on the previously defined intensity estimation and activity recognition tasks, comparing different base- and meta-level classifiers, is presented in [32, 33]. Overall, the best performance is achieved with boosted decision trees and \(k\)-nearest neighbors. However, the boosted decision tree classifier has further benefits: it is a fast classification algorithm with a simple structure and is thus easy to implement. These benefits are especially important for aerobic activity monitoring applications, since they usually run on mobile systems with limited computational resources. Therefore, boosted decision trees are applied in the classification step of the DPC for the rest of this section.

figure a

One of the key challenges identified by the benchmark is the necessity to improve existing algorithms to achieve good performance results on complex activity monitoring classification problems. Therefore, a novel boosting method called ConfAdaBoost.M1 is presented here. ConfAdaBoost.M1 (cf. Algorithm 1) is a confidence-based extension of the well-known AdaBoost.M1 algorithm. It is a direct multiclass classification technique, keeping the algorithmic structure of AdaBoost.M1. The main idea of ConfAdaBoost.M1 can be described as follows. In the training part of the algorithm, the confidence of the classification estimation is returned for each instance by the weak learner (line 5). These \(p_{ti}\) confidence values are used when computing the error rate of the weak learner (line 6): the more confident the model is in the misclassification the more that instance’s weight counts in the overall error rate. Moreover, the \(p_{ti}\) confidence values are used to recompute the weights of the instances. The more confident the weak learner is in an instance’s correct classification or misclassification, the more that instance’s weight is reduced or increased, respectively (line 12). Finally, the confidence values are used in the prediction part of the algorithm: the more confident the weak learner is in a new instance’s prediction the more it counts in the output of the combined classifier (line 21).

Table 2. Comparison of ConfAdaBoost.M1 to common boosting algorithms: test error rates [%] on the PAMAP2 activity recognition and intensity estimation tasks.
Full size table

The ConfAdaBoost.M1 algorithm has been evaluated on various benchmark datasets, comparing it to the most commonly used boosting techniques. Results achieved on the defined activity recognition (PAMAP2_AR) and intensity estimation (PAMAP2_IE) classification problems are shown in Table 2. It is clear that ConfAdaBoost.M1 performed significantly best among the compared algorithms. For example, on the activity recognition task the test error rate was reduced by nearly \(20\,\%\) compared to the second best performing classifier. A more detailed description of ConfAdaBoost.M1 and further results of its thorough evaluation can be found in [29].

Personalization of Physical Activity Recognition. The benchmark results on the PAMAP2 dataset show that although good overall performance is achieved on various activity monitoring tasks, the individual performance of the included subjects varies a lot [32, 33]. Therefore, personalization approaches are highly encouraged, thus to adapt a general activity monitoring model to a new user. This has become a topic of interest recently, suggesting personalization either in the feature extraction or classification step of the DPC. Drawbacks of existing approaches are that either the general model is simple (allowing only low performance on complex classification tasks) or too complex for mobile applications, resulting in unfeasible computational costs.

This section presents a novel general concept of personalization, applying it in the decision fusion step of the DPC. In this concept the general model consists of a set of \(S\) classifiers (experts), all weighted the same (\(w_i=1\), \(i=1,\dots ,S\)). Using new labeled data from a previously unknown subject, only the weights of the experts are retrained, the classifiers themselves remain the same. To show that this concept is a valid approach for personalization, different methods based on the idea of weighted majority voting are applied to increase the performance of the general model for new individuals. The baseline performance is given by majority voting (MV), thus when no retraining is performed. Using a set of \(N\) labeled samples from the new subject, three existing approaches are applied to retrain the general model: weighted majority algorithm (WMA), randomized weighted majority algorithm (RWMA) and weighted majority voting (WMV). Moreover, based on the proposed general concept, a novel algorithm called dependent experts (DE, cf. Algorithm 2) is introduced. The main idea of the DE algorithm is that the confidence of an expert’s prediction depends on the decision of all other experts. Therefore, the result of training the weights is a matrix of size \(SC\) (W, line 13), where \(w_{i,c}\) stands for the weight of the \(i^{\text {th}}\) expert when the majority vote of all other experts is the class \(c\) (defined as the performance rate of the \(i^{\text {th}}\) expert on this subset of samples, cf. line 8–10). This way, DE is more flexible compared to existing algorithms: it supports the case when an expert is performing good on some classes, but poorly on others.

The described general concept of personalization and the novel DE algorithm have been thoroughly evaluated on the PAMAP2 activity recognition task, using the LOSO evaluation technique [34]. The results show the validity of the proposed methods: compared to MV, the overall performance measures and especially the lowest individual performance increases significantly. Moreover, the new DE algorithm clearly outperforms all other methods and is thus a very promising approach for personalization. Since the presented algorithms are computationally not intensive, they are feasible for mobile activity monitoring systems. Finally, the proposed personalization approach requires less interaction from a new user than existing solutions and has a short response time [34].

figure b

3.3 Strength Exercise Monitoring

Different systems and methodologies for monitoring and supervising home-based motor retraining and coordination exercises have been proposed during the past years. See [25] for a thorough review of wearable sensors and systems for rehabilitation applications. Examples of rehabilitation solutions that have entered into the market are Hocoma’s ValedoMotion [14] and CoRehab’s Riablo [8]. Using few wearable IMUs, both system monitor specific body parts, such as back, knee, or elbow, with respect to range of motion and use gamification techniques to motivate the user.

Current video games include feedback based on wearable motion or external vision sensors in order for users to follow fitness exercises of general interest. While such gaming systems are motivating and can have a positive effect on strength, balance, and overall fitness, the considered parameters are undocumented leading to a lack of proper monitoring and helpful feedback. Moreover, the available systems cannot be personalized for users with specific needs and individual limitations and their use in frail populations has led to injuries as reported in a recent survey [38]. Finally, external vision sensors suffer from the line-of-sight problem and therefore restrict the set of available exercises to those which allow full visibility of the user in the relevant plane.

To summarize, previous work has mostly focused on a single body joint rather than providing a flexible solution for the whole body. Moreover, it has concentrated on the motivational aspect of the system rather than on developing sophisticated monitoring methodology. Therefore, this section focuses on a recently developed methodology, which takes both exercise load and technique into account and stands out due to the complexity of evaluation parameters, the inclusion of the complete body, and its generic concept with inherent personalization.

Objectives and Requirements. The aim of the strength exercise use case is to guide a user through a training session and to provide valuable online feedback in order to ensure positive training effects and prevent injuries through correct exercise execution. For this, the exercise load, as well as, the performed movement have to be monitored and evaluated. The latter includes verifying that the muscles loaded during the exercise are the targeted ones and that the range of motion and the assumed postures are correct. Monitoring these parameters requires, in contrast to the aerobic activity use case, short-term, but accurate tracking of relevant body segments.

The following paragraphs describe the technical realization of these requirements in terms of the back-end components depicted in Fig. 1. Data collection addresses the hardware platform, as well as, full-body motion tracking, while data processing encapsulates methods for: (1) to learn and then recognize motion patterns (single exercise repetitions) in continuous motion data streams; and (2) to compare segmented patterns to previously learnt reference motions and to evaluate their execution in terms of the above mentioned parameters.

Data Collection. Strength exercises are monitored over a short period of time, typically during training indoors and require accurate tracking of all involved body segments. Therefore, the hardware system for this use case is based on a stationary processing and display infrastructure (e.g., a laptop and a television) and a comparably complex wearable inertial sensor setup.

Any commercially available IMU, providing sufficient measurement quality can be in the system. While the latest generation wireless sensors, such as [39, 46], are rather costly and obtrusive due to their form factor, recent developments in sensor miniaturization enable low-cost and light-weight solutions [36].

The number of sensors, the sensor positioning, fixation, and calibration, is a trade-off between ease of use and data accuracy. The latter receives more emphasis here compared to the aerobic monitoring use case. To precisely capture the user’s movements, it is typically assumed to have one IMU on each major body segment that should be monitored. Moreover, sensors should be placed on bones, ligaments, and between muscles in order to be unobtrusive and limit the skin and muscle motion artifacts. Furthermore, an easy, fast and repeatable fixation method is required that neither allows for too many degrees of freedom nor is too size-dependent.

While previous systems emphasize flexibility and are mostly based on Velcro straps on top of the normal cloths [8, 14], the system in focus here, the solution proposed in [6], uses a modified sports suit with pre-defined sensor fixation points in order to reduce the burden on the user to remember the correct positioning. Moreover, an interactive, but easy-to-perform calibration procedure is used to improve data accuracy. Recent developments in the direction of smart garment with highly integrated miniaturized sensors provide a promising future platform for the considered application [36].

Fig. 3.
figure 3

Functional upper body model with indicated IMU placement (red cubes) (Color figure online).

Full size image

A conventional approach to body motion tracking is to determine the joint angles and angle kinematics by comparing the IMU measurements (accelerations, angular velocities, and magnetic fields) to predictions based on a biomechanical body model using model based sensor fusion. This biomechanical model is typically a functional model consisting of rigid bodies and joints, such as the upper-body model illustrated in Fig. 3 in relation to the sensor positions. To capture more detailed motions or additional body parts, the model complexity can be increased by including additional segments and respective IMUs. While commercially available inertial motion capturing systems based on the aforementioned type of IMUs [40, 47] provide rather closed solutions, dedicated implementations of the underlying method as described in e.g.,  [10, 21, 28] provide more flexibility. In particular, [21] describes a generic method for tracking arbitrary kinematic chains based on IMUs.

The inertial motion capturing system used in this section is based on the model illustrated in Fig. 3, while assuming the same structure also for the lower body. The full-body model consists of ten rigid bodies (torso, pelvis, upper-arms, forearms, upper-legs, and lower-legs) connected by anatomically motivated restricted joints. The orientations of torso and pelvis, as well as, the shoulder and hip joints are modeled with three degrees of freedom, while elbow and knee joints are modeled as pivot joints with two degrees of freedom. Hence, in total, \(26\) angles are available for data processing.

Due to a lack of commonly available datasets for the type of application described here, a new dataset has been created using this setup. This dataset is particularly interesting, since it has been generated in the context of a clinical study with an extremely relevant target group of elderly people between the ages of \(55\) and \(86\). The dataset contains motion tracking data (\(26\) joint angles at \(100\) Hz) from \(30\) participants performing \(10\) to \(13\) different upper and lower body exercises. These exercises were performed once under the supervision of a physical activity teacher and once autonomously. Labels indicate the start and end of each exercise repetition. From the \(30\) participants, ten were fit and healthy, ten were cardiac patients and ten suffered from upper or lower body functional disabilities. Hence, this dataset is suitable for evaluating personalized monitoring methodologies and will be used in the rest of this section for evaluation purposesFootnote 1.

Data Processing. From a technical point of view, monitoring both exercise load and performed movement requires to automatically detect single exercise repetitions and to accurately evaluate each repetition with respect to certain parameters. A promising concept for achieving personalized monitoring is to learn personalized exercise models from example executions of individual users. This is illustrated in Fig. 4: During a teach-in phase, training data is collected from a user, while he/she performs a certain number of exercise repetitions (per exercise), e.g., under supervision of a physical activity specialist. The training data is then used to automatically construct a personalized exercise model, which serves as gold standard during the trainer mode, i.e., during online exercise monitoring. This generic reference model concept not only enables personalization, but it also provides independence of a fixed exercise selection with pre-defined parameters for each exercise.

Fig. 4.
figure 4

Concept for personalized strength exercise monitoring.

Full size image

Subsequently, the technique to automatically generate a personalized exercise model from training data is described. This includes finding all repetitions of the performed exercise in the recorded sequence and then creating a statistical model from the detected repetitions. Afterwards, it is explained, how this model is used to detect and evaluate motion cycles during online monitoring.

Teach-in Mode. The following paragraphs describe a fully automated method for reliably extracting known numbers of exercise repetitions within continuous motion sequences. Technically, this corresponds to the problem of detecting motifs in multivariate training sequences, which is also referred to as unsupervised motif discovery. A motif is here a recurring motion segment, representing one repetition of strength exercise execution. The detected segments can then be used to generate a personalized statistical model, which can serve as reference for both online motion cycle detection and evaluation.

In the following, the different steps of motif discovery will be described. First, the dimensionality of the motion data is reduced. Figure 5a shows an example recording of the \(26\) angles for the full body. Based on the assumption that the most moving joints contain the most relevant information, the angles with highest variances are extracted (see Fig. 5b). Let

$$\begin{aligned} C_{i} = \{ c_{i,1}, c_{i,2}, \cdots , c_{i,n}\}, \end{aligned}$$

with \(0\le i<b\), be a time series of angle values for joint angle \(i\), where \(b\) is the number of tracked angles and \(n\) the length of the sequence. Then, each channel with \(\mathrm{var}(C_{i}) > \mu _{var}\) is considered to be relevant, with

$$\begin{aligned} \mu _\mathrm{var} = \frac{1}{b} \cdot \sum _{i=0}^{b} \mathrm{var}(C_{i}) . \end{aligned}$$

Since, in this case, the length of the pattern (one exercise repetition) is unknown, the second step is to estimate a suitable window size \(w_{\text {est}}\), i.e., the length of one motion cycle. This is an extension to most previous approaches, which are based on a predefined window size [22]. Here, the windows size depends on the sampling rate of the system and is measured in number of samples.

Based on the assumption that the repetitions in the training sequence are performed consecutively with roughly the same speed, a dominant frequency should be present in the signal. This can be extracted using the combined power spectral density (PSD) [44] (cf. Fig. 5b). The window length \(w_{\text {est}}\) is then initialized as the wavelength of the dominant frequency, \(w_{\text {est}} = \lambda = \frac{v}{f_{\text {dominant}}}\), with \(v\) being the sampling rate; i.e., here \(100\) Hz.

Fig. 5.
figure 5figure 5

Consecutive steps of the motif discovery on a sample teach-in sequence: (a) plots 26 joint angles recorded during a teach-in session for one exercise; (b) illustrates the reduced motion tracking signal, i.e., the most moving joint channels (top) and the PSD (bottom), where the dominant frequency is marked with a red circle (estimation of \(w_{est}\)); in (c), the annotated area shows the result of the motif candidate (green) selection, as well as the finally selected motif (red) (Color figure online).

Full size image

The next step detects the motif candidates. For this, an extended version of Minnen’s method [22] parametrized with \(w_{\text {est}}\) is used. The method collects overlapping sub-sequences, \(S_{i}\), of length \(w_{\text {est}}\) from the training signal, \(S\), and determines the \(k\)-nearest neighbors for each subsequence as \(\mathrm{kNN}(S_{i})= S_{i,1\dots k}\). Here, \(k\) is the predefined number of repetitions. In order to reduce the sensitivity to local time shift and slightly varying execution speed, dynamic time warping (DTW) is used as distance measure. A real motif should have at least \(k\) similar sub-sequences. Hence, in order to find good motif candidates, for each subsequence, \(S_{i}\), the distance density is estimated as the reciprocal of the distance to the least similar neighbor \(k\): \(\mathrm{den}(S_{i}) \propto \frac{1}{\mathrm{dist}(S_{i},S_{i,k})}\). The motif candidates, \(cand_{i}\), are then identified as the local maxima of the densities among their \(k\) nearest neighbors:

$$\begin{aligned} {\text {maxima}}(S_{i}) = \{ S_{i} :\ \forall \ S_{i,j} \ \mathrm{den}(S_{i}) > \mathrm{den}(S_{i,j}) \}, \end{aligned}$$

where \(j = [1, k]\). Motif candidates are highlighted in Fig. 5c.

In the next step of the algorithm, a model for each candidate is generated and used to segment the signal. As most of the learning approaches fail, if there are only few training samples available, either constructed models [42] or template-based approaches are feasible. Here, a template approach, based on the online dynamic time warping (ODTW) [17] is described. The motif candidate is chosen as the template for the DTW and its \(k\)-neighbors are used for defining the cost threshold.

Finally, the candidate, which model provides the best signal segmentation, is chosen as the motion motif. As criteria, the difference between the number of segmented patterns and the known number of executions in the training sequence, as well as, the average normalized DTW costs are used. In Fig. 5c, the selected candidate is marked red.

The chosen motif and its nearest neighbors can now be either used to generate a class template from a set of the best templates, e.g., extract the templates from the nearest neighbors that have the best minimum inter-class DTW distances [17], or they can be used to generate a hidden Markov model (HMM) as proposed in [42]. Both approaches are suitable for an online real-time segmentation.

The proposed motif discovery method was evaluated on the previously mentioned dataset in terms of precision, recall, and overlap. Precision is defined as the fraction of segmented exercise executions that are relevant, while recall is defined as the fraction of correctly retrieved executions. A segment, i.e., a motion cycle, is considered as correctly retrieved, if it overlaps with the ground truth segment exceeds 30 %.

Table 3. Exemplary motif discovery results averaged over all performed exercises in terms of precision, recall, and percental overlap.
Full size table

Table 3 exemplifies the experimental results for selected participants represented in the dataset, averaged over all performed exercises. A more detailed analysis of the results is given in [43]. In the following the methods for generating personalized models for real-time, online motion cycle detection, as well as, for motion cycle evaluation are introduced.

Model Construction and Motion Cycle Detection. Based on the motion segments extracted in the teach-in mode, a method for constructing an HMM for a motion pattern, is discussed in this section. The HMM representation is chosen for two reasons: (1) it naturally takes variations in motion into account by allowing for time-warping and has thus been successfully applied in domains such as speech, gesture, and handwriting recognition; (2) standard algorithms, such as the short-time Viterbi algorithm [7] can be applied for online, real-time monitoring.

The observation probabilities of the HMM are modeled using Gaussian mixture models (GMM), as illustrated in the left plot in Fig. 6. Here, the different joint angle components of the multivariate signal are handled separately.

Let \(RM\) be the set of reference motions recorded for one exercise performed by one individual, during the teach-in phase. Now, a model \(M_{RM}\) is learnt from the reference motion \(RM\). Since traditional parameter estimation methods for HMMs, such as the Baum-Welch algorithm, typically fail when applied to too few training examples, a simple construction algorithm is applied to capture the characteristics of each reference motion \(RM_{i}\). This algorithm builds a HMM with left-right topology, which is a wide-spread approach to model time-varying sequential data [26]. Self-transitions and skip-transitions are added to allow for a faster and slower execution of the pattern. The number of states, \(N\), is chosen as half the average sample length \(l_{\text {avg}}\): \(N = \lceil \frac{l_{\text {avg}}}{2} \rceil \) of the reference motion patterns \(RM\). For each state, \(ST_{i}\), a GMM is then trained using an expectation-maximization algorithm on all respective elements of \(RM_{i}[j:j+l_{\text {avg}}]\). Thus, each segment is described by one normal distribution \(\mathcal {N}(\mu _{j}, \sigma _{j})\).

The HMMs obtained during the personalized model creation enable online detection of the represented reference motion within continuous motion data by utilizing the short-time Viterbi algorithm [7]. In general, the Viterbi algorithm computes the most likely path of states given a sequence of observations. Here, the observations are given by the continuous joint angles as streamed by the data collection component. Thus, the algorithm can determine, to which state, respectively frame, of the reference motion the current motion matches. If the probability of the Viterbi algorithm is below a defined threshold, the current observation is considered represent an incorrect motion. The motion cycle detection immediately allows for counting exercise repetitions and deducing their duration. Whenever a complete motion cycle has been detected, the detailed evaluation starts as will be detailed below.

Motion Cycle Evaluation. According to the system requirements, it is fundamental to evaluate the load of the exercise, the muscles that work, as well as, the posture assumed during the exercise in order to ensure effectiveness and safety. Translating these constraints into objective data that can be derived from the measured motion data resulted in the following criteria: For movement load, the exercise intensity is quantified by the number of repetitions, the movement speed, the movement amplitude, and the movement smoothness. Whether the muscles that work are the correct ones is evaluated based on the axes of rotation during motion. Finally, for safety issues, the posture is evaluated based on a number of fixed distances or angles to be kept when performing the movement. This could, for instance, be the distance between the feet during squat exercises, or the angle at the pelvis during push-ups.

Fig. 6.
figure 6

HMM for one angle of the motion signal.

Full size image

The number of repetitions and their duration are given by the motion cycle detection step described above. For the other criteria, an algorithm has been proposed in [6], which evaluates each detected motion cycle using the model constructed during the teach-in mode as reference. The different steps of the algorithms are the following. First, different angles and distances that must be respected during the movement in order to avoid injuries are computed and compared with those obtained during the reference movement. Afterwards, the principal rotation axis is computed for the current cycle at each joint. The principal rotation axes are then compared to the ones obtained during the reference movement. Using the same formalism, the rotation amplitudes are also compared. Finally, the number of local extrema of the time derivative of the joint trajectory (i.e., its velocity) that has the greatest range of motion during the movement is evaluated and compared in order to determine movement smoothness. The procedure is illustrated in Fig. 7. The movement duration used to evaluate velocity, the pose (fixed angles and distances), and the rotation amplitudes of the movement to evaluate should not differ by more than a certain threshold from the reference model. The principal rotation axes should not deviate more than a certain threshold from those obtained from the reference. For the smoothness, the same number of extrema has to be found, since any other number of extrema indicates a deviation from the prescribed movement, e.g., in the form of a parasite movement or a break in motion flow during the execution of the exercise. For providing online user feedback, if any of these above mentioned criteria are not met, an alert can be generated and sent to the user interface, which translates this into explanatory feedback. The motion cycle evaluation concept is summarized in Table 4. Moreover, a technical evaluation of the proposed algorithm in terms of a confusion matrix for the considered parameters within a small-scale study can be found in [6].

Fig. 7.
figure 7

Motion cycle evaluation: Detected motion cycles are evaluated separately by comparing them to a reference motion. The cycles overlaid with a red area show a significant deviation from the reference movement (illustrated as dashed red lines) (a), either in amplitude (b) or in the number of extrema (c). The green area indicates a correctly performed cycle (Color figure online).

Full size image

4 Open Problems and Future Outlook

This chapter has outlined a platform for personalized physical activity monitoring by means of wearable sensors and has showcased different possible use cases. Even though promising results have been shown that indicate the potential of the technology, there are still open problems and challenges to be solved before this technology can be widely applied.

In the aerobic exercise monitoring use case, it should be investigated how well the developed methods and algorithms perform with different user groups. The evaluation on a publicly available physical activity monitoring dataset — including only young, healthy adult subjects — indicate good performance results. However, the ability to generalize the developed approaches with significantly different user groups (e.g., elderly) remains an open question. Moreover, it is also planned to investigate the effect of increasing the number of known (thus in the training included) other activities, with the goal to improve even more the robustness with respect to unknown other activities while sustaining the high performance regarding the basic activity classes of interest. Furthermore, although the mobile aerobic activity monitoring system in its current form (using small wireless sensor units and a smartphone as mobile control unit) is as unobtrusive as possible with today’s technology, its acceptance amongst different user groups needs to be evaluated in a user study.

Table 4. Motion cycle evaluation concept: The fixed angles and distances are provided by physical activity experts. The other measures are directly deduced from the movement of reference. The values \(x\) and \(\theta \) are parameters of the algorithm.
Full size table

In the strength exercise use case, for instance, the detailed motion capturing needs to be more robust and the wearable sensory equipment even less obtrusive to be reliably deployed in the users’ homes. Current developments in miniaturized sensors and smart clothes are fully in line with the latter requirement and open up for new possibilities. Moreover, stationary vision and depth sensors used in current gaming consoles could be fused with wearable motion sensors in order to create synergies and increase precision and robustness or reduce the number of required wearable sensors. A purely vision-based approach, however, is not feasible for the type of motions performed, due to the line-of-sight problem.

Another challenge is posed by the machine learning algorithms used to learn, segment, and evaluate motion cycles of arbitrary exercise motions. Today, these algorithms require engineering know-how to some extend to tune various parameters and thresholds. Here, further experiments and the development of data-driven parameter selection methods are crucial in order to improve the usability of such technologies for e.g., health care professionals.

A last aspect to mention for the strength exercise use case is the fact that recognition algorithms sometimes fail. This raises the question how this should be handled by both the monitoring system and the user interface. Here, in particular false negative motion cycle detections or false positive incorrect motion detections could decrease the motivation of the user rather than providing support. Furthermore, a system accepting incorrect motions could be even more dangerous and could encourage the user to hurt himself. A forgiving user interface and the possibility for online learning based on some type of feedback from the user could be subject of future research.

Even though open problems still exist, monitoring technologies, such as the ones presented in this chapter, have taken a big step forward and in view of today’s societal challenges, the aging society and the aging workforce, it is to be expected that mobile health technologies will gain even more importance and enter various application fields from personal health and fitness over work-life-balance support to human factor research.

Reading

The generic platform concept for physical activity monitoring using wearable sensors, as well as the two implemented use cases have been presented in the previous Sect. 3. Especially the technical requirements in terms of the hardware platform and monitoring methodology have been addressed. However, in order to design such a system it is inevitable to consider end user requirements and to evaluate its overall usability. Particularly, when designing for diversity (e.g., elderly population) additional requirements have to be considered.

Targeting the elderly population various aspects and effects of aging on mental and physical health and fitness have to be considered. Among others the book

  • K. Berger. The developing person: Through the life span. Worth Publishers, 2008

describes the cognitive changes with aging (decreased ability to perceive a high amount of information at the same time, decrease of memory), the physiological changes (decrease of sensory abilities, e.g., vision and hearing, decrease of movement accuracy and coordination) and the ability to deal with recent technology. The cognitive and the physiological changes should be taken into account during the conception of the user interface and the conception of the wearable sensors. Regarding the user interface, the quantity of information presented to the user should be reduced to the most useful and simplest form and should be presented by different sensory means (visual, auditive). The interaction with the system, as well as the manipulation and the fixation of the wearable sensors, should not require any fine movements. Finally, the user interface should be integrated into a system familiar to the user in order to limit the required learning of unknown technology. For a detailed description regarding the above mentioned requirements the interested reader is referred also to the following books

  • A. D. Fisk, W. A. Rogers, N. Charness, S. J. Czaja, and J. Sharit. Designing for older adults: Principles and creative human factors approaches. CRC press, 2009

and

  • A. Sears and J. A. Jacko. The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. CRC Press, 2007

Moreover, recent advances in the miniaturization of sensors have made it possible to realize a great variety of new applications and open up new possibilities. However, this vast amount of sensor data has to be captured, stored and analyzed. Finding patterns and trends in this data is a challenging task. Witten et al. presents in

  • I. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques, Second Edition. The Morgan Kaufmann Series in Data Management Systems. Elsevier Science, 2005

a description of the Weka toolkit, along with a thorough foundation for the machine learning concepts the toolkit uses, and practical advice for using the different tools and algorithms. Weka is a collection of machine learning algorithms for data mining tasks. It includes tools for data pre-processing, classification, regression, clustering, association rules, and visualization. It is also well-suited for developing new machine learning systems.

A comprehensive and up-to-date introduction to the theory and practice of artificial intelligence is given by Russell and Norvig in

  • S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall series in artificial intelligence. Prentice Hall, 2010

A great introduction to ubiquitous computing is given by Krumm in

  • J. Krumm. Ubiquitous Computing Fundamentals. Taylor & Francis, 2009

This book covers the contributions of 11 of the most prominent researchers in the field of ubiquitous computing. Based on the categories systems, experience, and sensors the authors describe various research topics in the field of ubiquitous computing.

Finally, pattern recognition from the Bayesian viewpoint is addressed in the book

  • C. Bishop. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, 2006

This book does not require any previous knowledge of pattern recognition or machine learning concepts. Furthermore, it includes a self-contained introduction to basic probability theory.