Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Human motion science and motion analysis have a long history [1]. The Renaissance has been a fruitful period in understanding the underlying mechanics of the human body, and it is in the 17th century that Isaac Newton, and later Lagrange, have really enabled to write the equations of motion as we know them. The 19th century and the 20th century have then enable to measure what these equations were expressing, with the development of photography and motion picture and the famous work of Muybridges and Marey to cite only the most famous.

The last decade of the 20th century and the 21st century have been marked by the developments of motion capture technology, with faster capture rates, higher measuring precision and resolution. Not only can the kinematics of a motion be measured but its dynamics too with the use of force sensors and force-plates. These systems have paved the way for numerous applications and developments and are now commonly used in animation, sport science, motor rehabilitation and robotics [2–4]. On the other hand dance and performance professionals have developed powerful systems to annotate, record and reproduce extremely complex choreographies and motion sequences [5]. From the kinematics information, systematic annotation can be done [6], in addition from a set of notations, motion kinematics can be generated and therefore used for animation [7]. These offer an extremely powerful mean to generate and record motion of any entities, human or not and the annotation are crucial to understand these motions. Labanotation (also known as Kinetography Laban) offers a compact and powerful notations system for motions. Each motion can be described with a number of defined elements just like a score in music. Labanotation treats of the dynamics which is expressed by the body of a performer, and the way the performer interacts with the environment which is a difficult concept. However Labanotation does not quantify or measure these factors. Yet the dynamics is an important aspect of motion since entities are bound to external forces: gravity, and then interact with their environment to create motion, to generate more or less force, express something, convey energy. The sole kinematics cannot describe these interactions and thus incompletely describes motions. Our research offers a mean to quantify and bridge the gap between classical dynamics analysis in biomechanics and in robotics and in annotations in Labanotation.

We present a general framework to analyze and classify motions and in particular dynamics in order to highlight some specific value: motion, emotion, or mood that can be used to systematically define dynamics as defined in Labanotation [8]. Our framework is based on the definition of a feature vector of the data and of a decomposition in principal components (PC). The decomposition in the PC space is used as the base of our classification and recognition algorithm.

The chapter is organized as follows: in Sect. 2, we first describe the terminology used to describe a motion and its dynamics and how it relates to or differs from Labanotation developed to analyze and record motion in dance and more generally performances. Section 3 presents the feature vector analysis using principal component analysis (PCA). Section 4 presents concrete examples using torso data and contact forces. Section 5 provides some extension of the results and proposes to use dynamics information for emotion and mood classification. Section 6 concludes this chapter by offering some perspectives for future applications and developments.

2 Equation of Motion and Terminology

The equation of motion (1) in its most general form [9, 10] provides a simple mean to understand the time variant relationship between what is called in Biomechanics [1] and Robotics [11] a motion: displacement and orientation of a reference, joint angles of the articulated system; the forces and moment of forces applied on the environment and the forces generated by the muscles and transmitted through the joint torques. It also makes use of time invariant such as the body segment lengths and the dynamic properties of these segments, such as masses and inertia.

$$ \left[ {\begin{array}{*{20}c} {\mathbf{0}} \\ {\varvec{\Gamma}} \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} {\varvec{J}_{\varvec{b}}^{T} } \\ {\varvec{J}^{T} } \\ \end{array} } \right]\varvec{F} = \left[ {\begin{array}{*{20}c} {\varvec{M}_{\varvec{b}} } & {\varvec{M}_{\varvec{c}} } \\ {\mathbf{0}} & {\varvec{M}_{\varvec{c}} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {\ddot{\varvec{\textit q}}_{\varvec{b}} } \\ {\ddot{\varvec{\theta}}} \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} {\varvec{C}_{\varvec{b}} } & {\varvec{C}_{\varvec{c}} } \\ {\mathbf{0}} & {\varvec{C}_{\varvec{c}} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {\dot{\varvec{q}}_{\varvec{b}} } \\ {\dot{\varvec{\theta }}} \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} {\varvec{G}_{\varvec{b}} } \\ {\varvec{G}_{\varvec{c}} } \\ \end{array}} \right] $$
(1)

where M c , C c , G c , M b , C b , and G b , are the inertia, Coriolis and gravity matrices calculated at the articulated chain (subscript c) and at the floating base level (subscript b), respectively. \( \dot{\varvec{\theta }} \), \( \ddot{\varvec{\theta}} \), and \( \varvec{\varGamma} \) are the joint velocity, acceleration and torque vectors, respectively. J T is the Jacobian transpose matrix mapping the external forces to the joint space and \( \varvec{J}_{\varvec{b}}^{T} \) is the Jacobian transpose of the matrix mapping the external forces to the floating base frame. Finally, \( \dot{\varvec{q}}_{\varvec{b}} \), and \( \ddot{\varvec{\textit q}}_{\varvec{b}} \) are the base Cartesian velocity and acceleration vectors, respectively.

Depending on the available measures and the region of interest for the human motion analysis different approaches can be considered. Some studies [12, 13] consider only the movement of the extremities and or the torso since they are of low dimensions, what is often called the task space in robotics [11]. Labanotation system describes the displacement of weigh (weight transfer) and the displacement of the parts of body, and the way of its displacements in physical space [5] (=task space). Most of the studies on human motion consider the movement at the joint level [14–16], what is often called the joint space in robotics [11] Labanotation system segments a human body by body signs which specify limbs, joints, areas, and surface of the body [17]. Concerning the motion of limbs, though space measurement signs specify the degree of flexion of limbs, a movement does not be considered by only change of the degree of joints in Labanotation. Both cases in robotic and in Labanotation, data are referred as kinematics and are actually related [11–18].

One other interesting aspect is the contact with the environment and how the body generates force to interact with it and receives force from it [19, 20]. In Robotics it is called external forces [11]. Finally, the computational tools of Robotics [11] and Biomechanics [1] allow to also consider the inner forces acting on the body such as the joint torques and the muscle activity and muscle forces. In Labanotation, a notion of dynamics seems similar to these concepts of external and inner force in robotics. Dynamics in Labanotation links to the quality of the movement that varies according to flow of energy and force, different way of using time and space during a performance [8]. However, Labanotation does not measure external force which can affect human motion, nor inner force. Labanotation categorizes different accentuation, intensity or muscular tension observable during the execution of a movement by strength measurement signs (dynamics signs) [5]. In Laban Movement Analysis (LMA), a concept of Effort, which is one of the four components of LMA, focuses on clarifying a motion dynamics [21]. Laban Movement Analysis is a method to analyse, observe and explore human motions, and it differs from Labanotation (Kinetography Laban) which aims to translate movement process taking place in four dimensions (three dimensional spaces and time) into signs written in two-dimension [18]. Effort analyses a human dynamic and its relations with personality or psychological reactions such as emotion, feeling, and thinking in a descriptive and indicative way by using a visual support, but Effort does not take account into how muscular forces generate such dynamics.

In this study we consider the dynamics in the general form of the equation of motion, when the word refers to the Labanotation dynamics it is explicitly mentioned, since this term in the Labanotation refers to the left hand term of Eq. (1) and eventually also includes G b and G c according to [8].

3 Feature Vector Analysis Using Principal Component Analysis (PCA)

In this section we present a framework to classify time series of data: here motion data using what is called a feature vector and introduced in Sect. 3.1. The feature vectors are then analyzed using principal component analysis (PCA) to obtain clouds of points. The points’ proximity indicates resemblance in the motion data, and can form clusters of similar motion data as developed in Sect. 3.2. These clusters can then be used to analyze furthermore the data, develop recognition algorithms and so on as presented in Sect. 3.3.

3.1 Construction of the Feature Vector of a Data

A feature vector is a vector that contains characteristics that could quantify various data q i : kinematics, dynamics, in the joint space, in the task space, contact forces, muscle activity… It is obtained by computing the auto-correlation matrix of the considered data q i [22]. The auto-correlation matrix O i (l) of a given data-set is given by Eq. (2). l is a time constant difference, here we set l = 2, because we can reflect the information of the data change (velocity for example). Then we arrange the elements of O i (l) into a single column vector. The result is called the feature vector for the given motion data q i for an iteration i. For each motion repetition or each new motion i the feature vector is computed.

$$ {\mathbf{O}}_{i} (l) = \frac{ 1}{{T_{i} - 2}}\sum\limits_{k = l + 1}^{{T_{i} }} {{\mathbf{q}}_{i} [k ]{\mathbf{q}}_{i}^{\text{T}} [k - l ]} $$
(2)

3.2 Principal Component Analysis of the Feature Vector of a Data

Principal Component Analysis (PCA) of the obtained feature vectors provides information of the clustering properties of the data. A training data-set is created with a few exemplars of data. Consequently, it gives information on the possibility to discriminate a data-set from another data-set [22]. Applied to motion recognition, it means that it gives information on the differences and resemblances of different motion data-set; it allows discriminating between several motions, for which the algorithm was trained. This algorithm functions as an unsupervised learning algorithm, since data as data accumulates the PC space varies and clusters appear. Depending on the resemblances, points create clusters of various shapes, in the space of principal components, which are dense or scattered. It is also possible to find whether a motion belongs to the training data or not, and perform incremental learning.

Often the three dimensional space of the first three principal components is used because it allows a visualization of the dataset. The two dimensional principal component (PC) space can also be used if the cluster structure is clear enough using only the first two components. The shape of a cluster highlights data-set with similarities, while scattered points represent data-set with little similarity to each other.

3.3 Motion Classification Algorithm

The proposed recognition algorithm is based on the clustering in the PC space of the feature vectors [23]. In the feature vector PC space, we calculated a feature value using the center point of each cluster and the approximate straight line by the least-squares method of each cluster as shown in Fig. 1. It thus provides the center and the approximate shape of the cluster. When a test data needs to be compared with training dataset, it is projected into the PC space and the distance from each cluster center d ij as well as the distance to the linear regression h ij are calculated. The weighted sum of d ij and h ij gives the feature value S ij which is used to determine whether the test data belongs to one of the existing clusters (if so which one) or not. The smallest S ij allows to conclude that the test-data belongs to cluster Motion j. If it is not small enough then one can conclude that it belongs to none of the existing cluster and a new cluster can be created. The process is described in Fig. 2, where the test data belongs to the cluster “Motion 4”.

Fig. 1
figure 1

Feature vector in the PC space forming clusters of similar motions for four examplar of motion data (color of markers) repeated each five times (number of markers of each color). And barycenters (round marker) and linear approximations (straight line) for each cluster

Full size image
Fig. 2
figure 2

Concept of the proposed recognition algorithm using the barycenter and the linear approximation for each clusters in the PC space of the feature vectors. The distance from each cluster to the test-data provides a metric: feature value that allows to associate the data with a cluster (in that case Motion 4), or without any and is the base of our recognition algorithm

Full size image

The weights of the weighted sum can be chosen to give equal weight to the distance from the straight line and the center, or to favor one over the other depending on the shape of the cluster. For example a rather spherical cluster may give more weight to the distance to the center, while an oblong cluster may require more weight on the distance to the straight line. Using the training data to define the shape of the cluster can be used to generate automatically appropriate weights. These weights need to be recalculated any time a new test-data is inserted in the dataset since the addition of a new data point may change the global shape and size of the clusters.

4 Examples of Motion Classification and Motion Recognition

The feature vector can be obtained with any kind of data for any type of motion. It can be used to classify motions and there resemblance, it can be used to classify inner state that are embedded in a motion: moods and emotions. In fact it has been shown in the literature that motion carry more information than just the simple kinematics and dynamics, and these data relates to human inner states [24, 25]. In the following section we show some examples of utilization of our algorithm and the potential it can offer for automatic notations through quantification.

4.1 Using Kinematics Data in the Task Space: Gait

Gait is one of the most common task in daily life, and it is also the most important locomotion characteristics of human and humanoids. Gait is rich in information and it was shown extensively in our previous work that biometrics from gait is possible [23], as well as emotion recognition using the torso motion and the head inclination, with a simple similarity index [13]. Using the same dataset of 4 candidates and 5 repetitions of each emotional gait, which is extensively described in [13], the individual emotion recognition rate reaches 90 % in average. For further detail in the protocol and the results please refer to our works [13, 23] (Fig. 3).

Fig. 3
figure 3

Feature vector in the PC space for the four basic emotions in 5 gait trial for a given individual [13]. “Sadness” strikes out the rest of the emotions. The close up on the right shows that when sadness is not considered, each emotion also appears in a cluster, allowing for emotion recognition in gait

Full size image

4.2 Using Contact Force Data

The dynamics of the motion also contains rich information. Each motion has a dynamic signature that is characteristic with the way one interact with the environment and according to (1) it is possible to link the external contact forces and the motion directly. We propose to use the contact force information measured by force plates or insole sensors for example to classify motions with the algorithm proposed in Sect. 3. In our preliminary works [19, 24] we proved that it was possible to recognize motions using only the contact force measurement regardless of the person characteristics using a laboratory grade force-plate that provided the 3 forces and the 3 moment of forces. We now propose to use Nintendo wii balance boards that can only measure the vertical force and the moment of force in the horizontal plane. The algorithm is tested experimentally on the data of 5 subjects for 7 types of exercises motion.

4.2.1 Experimental Protocol

Figure 4 presents the type of motions that were performed on top of two Nintendo wii balance boards. This set of seven prescribed motions was chosen among a Japanese daily television exercise program (Radio Exercise). Five male candidates (mean age 24) were shown a video of the motions they were asked to perform, prior to the experiment. During the experiments, the same video was shown so that candidates can synchronize with the video. Each motion was repeated five times to insure enough training data and enough test data, in addition to the fact that a motion sequence may consist in repeating a few times the same motion. We measured the contact force information for each of the chosen seven sequences of motions noted M1–M7 as shown in Fig. 4.

Fig. 4
figure 4

The seven different types of movements recorded on top of the forceplate for five male candidates. Each motion was repeated five times, to provide sufficient training data and test data

Full size image

4.2.2 Results and Discussion

The recorded motions are automatically segmented [26]. The feature vectors were calculated with the vertical forces measured by the Nintendo wii balance board and the moment of force around the horizontal plane. In total 6 components. The classification results are given in Fig. 7. We use the exclusion method proposed in [23]. This suggests that with trained subject the repeatability would in-crease and that recognition would be much easier. With un-trained subject the size of the clusters increases. This can also be used as a quantification of training performance or motion repeatability.

Figure 5 show the PC space obtained for all the candidates and all the motions. Clear clusters appear in the PC space, while some data are more confused. The exclusion method automatically removes the data that are too far to narrow the search space of possible motions. The total average recognition rate reaches 75 %, as shown in Table 1. The confusion matrix given in Table 2, also shows that motions that have high similarities are more difficult to discriminate from each other, such as M3 and M6 that only differ from the direction of the bending. This suggests that some motions have poor repeatability: poor execution of the motion by the candidate for example. For example if one candidate moved his arm faster or slower, or with less amplitude in one motion, it is plotted at a point that is far from the motion’s cluster. This suggests that with trained subject the repeatability would increase and that recognition would be much easier. With untrained subject the size of the clusters increases. This can also be used as a quantification of training performance or motion repeatability in terms of dynamics.

Fig. 5
figure 5

Feature vector in the PC space for the seven motions repeated each five times, by five candidates. Both graphs represent the same results with a different view point

Full size image
Table 1 Summary of the successful recognition rate [%] for each motion using the proposed exclusion method
Full size table
Table 2 Summary of the confusion matrix (M1–M7) using the proposed exclusion method
Full size table

5 Application for Emotion and Mood Classification in Motion

5.1 Using Contact Forces

Using a similar experimental protocol as described in Sect. 4.2.1, candidates were asked to walk on top of the force plates and to change their mood while walking. Each data was taken 5 times. The obtained PC space is shown in Fig. 6. A clear cluster structure appears and the average recognition rate for all of the four emotions reaches 80 %. These results suggest that the contact forces with the environment and thus the dynamics contain also emotional information. This allows to quantify and classify dynamics data and thus will likely enable automatic classification and annotation of the dynamics in the Labanotation system and of the Effort in Laban Movement Analysis.

Fig. 6
figure 6

PC space obtained from the contact force information while walking with different moods: normal, sadness, anger, happy. Each form a cluster charateristic showing that emotions can be recognized through the way one interact with the environment dynamically

Full size image

5.2 Using Kinematics and Muscle Contractions

As explained in Sect. 2, in Labanotation the dynamics describes a wide range of variables and among them the muscular intensity plays an important role. Muscular information relates with the dynamics through the joint torques. Moods show a variety of physiological manifestations that can be measured with a diverse array of techniques. The physiological activities such as brain waves, skin responses, heart responses and muscle responses, reflect change of a mental condition. The information on the mood included in behavior is classified into nonverbal information, and is also included in behavior without necessarily being based on the intention of an agent [27]. With surface electromyography (EMG) it is possible to record the muscular activity of surface muscles and associate it with motion and emotion. In this section we present our findings regarding the relationship between shoulder and upper-arm muscular contractions and mood variations. Identification of emotion and mood using EMG information has been done with a variety of methods until now [28–31]. Most are based on use of facial muscles. Walking is one of the key actions when identifying the information on humans. It is known that human walking includes information that is specific to the individual and be affected by mood [13, 24, 25]. That is, it is thought that the EMG analysis of walking is effective in the identification of human mood. In this work, we made a subject walk in various mood states and answer psychological tests that measure the mood. We use two types of tasks: music listening (affective value scale of music (AVSM) and multiple mood scale (MMS) [32]); and numerical calculation for evoking different moods. It is expected that numerical calculation evokes either a more aggressive or frustrated mood upon failure, or self-satisfaction when successful, than listening to music. Figure 7 shows the experimental protocol of our experiment. In foreground space the task is listening to calm music (odd numbers in the green circles). In the background space the subject’s task which is either listening to music (uplifting and depressive) or doing the numerical calculation (even numbers in the green circles). Inserting a task of listening to calm music between each other tasks allow to stabilize the moods as can be seen from Fig. 7-right.

Fig. 7
figure 7

Experimental setup for the mood changing experiment. Left The experimental setup at College de France with the two working stations and the free space for walking in between. Right Principal component analysis of the mood scores of the psychological TDMS test at the different stages of the experiments. The number in the green circle corresponds to the number of occurrence. It shows that the mood changes with time during the experiment as expected

Full size image

Statistical features of EMG signals are calculated using Fast Fourier Transform and Principal Component Analysis. These statistical features are related with psychological test scores obtained from a two-dimensional Mood Scale (TDMS) [33], using regression analysis.

The two types of tasks are performed on two laptop computers. Moreover, the tasks are done by turns at two distant spaces in the room, and the subjects walk back and forth between them. After completion of each task, the subjects fill in questionnaires about their mood at that time, and then the subjects walk to the other space. The TDMS test consists of eight questions on a 6-point Likert scale, and can quantify the state of mind at the time of measurement. The result of TDMS is expressed as a score of “pleasure” and “arousal” (from −20 to 20). In other words, high pleasure indicates a comfortable and positive state, high arousal indicates an excited and active state. The test is filled in less than 1 min, thus it is suitable for temporal observation of mood changes. The subjects repeated that work 8 times. Subjects were not informed that the motion for interest in our study was walking to insure walking as naturally as possible.

Surface EMGs of the biceps, triceps, middle deltoids, and upper trapezius were recorded. The mean power frequency MPF obtained by Fast Fourier Transform (FFT) of the EMG data were generated for each muscles for the PCA.

$$ {\text{MPF}} = \frac{{\mathop \int \nolimits_{0}^{\infty } fP\left( f \right){\text{d}}f}}{{\mathop \int \nolimits_{0}^{\infty } P\left( f \right){\text{d}}f}} $$
(3)

where P is the power of the signal, and f is its frequency.

The PCA was performed for each frequency range over all the muscles. Finally a multiple linear regression analysis (MRA) was performed on the principal component score as an explanatory variable and the amount of variation in the TDMS pleasure score which is the difference between each pleasure score and the previous one as a response variable. By considering the score not using an absolute value but using a relative value, we can take into consideration the individual difference of the reaction to a stimulus.

5.2.1 Experimental Results and Discussion

As a result of the regression analysis as shown in Fig. 8, we can confirm a statistically significant positive correlation between the muscle activity and the arousal level (p < 0.001) with R = 0.59. This correlation is promising in using muscle information to predict moods. Moreover muscle contraction are related to more observable variable such as joint stiffness, allowing for quantification of mood/emotion through joint stiffness.

Fig. 8
figure 8

Experimental results. Left Mean power frequency of EMGs of muscles around the shoulders and psychological TDMS test scores of “pleasure” correlates. EMGs can be used to predict the mood state. Right Loading factors of the different muscles involved

Full size image

6 Conclusion and Perspectives

This chapter emphasizes on considering not only the kinematics information of a motion but also the dynamics information, and in particular the contact forces with the environment when analyzing human motion or simulating motions for animation or humanoid robotics. Indeed, the dynamics play an important role in robotics to control the stability and generate smooth and safe motions. In biomechanics it allows to understand the interaction of the human body with the environment and in Labanotation, dynamic is important criteria to observe and describe the way and the quality of movement and performance. The theory of Effort in Laban Movement Analysis treats exclusively a study of dynamic and explores its relation with psychological state with using a rigorous observation grid [21]. Using a quantification, a classification and a systematic analysis of the dynamics as presented here will allow to generating systematic dynamics annotations, both as understood in robotics [11] as well as in the Labanotation [8]. This systematic notations will help in bridging the gap of terminology of both communities and in providing a general framework and understanding that will benefit both communities. Our promising results in using contact forces, muscle activity and further joint stiffness to understand mood and emotions are also crucial to generate motions that can convey feelings, and understand these motions when performed by humans.

If some of these results are at a preliminary stage, we are confident that they will lead to outstanding outcome in human motion science and further to humanoid motion generation.