Classifying a Pirouette

Abstract

A pirouette is a familiar step in ballet. There are many types of pirouettes. We will focus on an en dehors (outside) pirouette from fourth position. The dancer pliés (does a deep knee bend) then straightens her legs producing both an upward force to get on the tip of her pointe shoe and a torque to turn about her axis of revolution.

7.1 Introduction

A pirouette is a familiar step in ballet. There are many types of pirouettes. We will focus on an en dehors (outside) pirouette from fourth position. The dancer pliés (does a deep knee bend) then straightens her legs producing both an upward force to get on the tip of her pointe shoe and a torque to turn about her axis of revolution.

In this chapter, we will classify pirouettes. Four dancers will each do ten double pirouettes, and we will use them to train the deep learning network. The network can then be used to classify pirouettes.

This chapter will involve real-time data acquisition and deep learning. We will spend a considerable amount of time in this chapter creating software to interface with the hardware. While it is not deep learning, it is important to know how to get data from sensors for use in deep learning work. We give code snippets in this chapter. Only a few can be cut and pasted into the MATLAB command window. You’ll need to run the software in the downloadable library. Also remember, you will need the Instrument Control Toolbox for this project.

Our subject dancers showing a pirouette are shown in Figure 7.1. We have three female dancers and one male dancer. Two of the women are wearing pointe shoes. The measurements will be accelerations, angular rates, and orientation. There really isn’t any limit to the movements the dancers could do. We asked them all to do double pirouettes starting from fourth position and returning to fourth position. Fourth position is with one foot behind the other and separated by a quarter meter or so. This is in contrast to fifth position where the feet are right against each other.

Each is shown at the beginning, middle, and end of the turn. All have slightly different positions, though all are doing very good pirouettes. There is no one “right” pirouette. If you were to watch the turns, you would not be able to see that they are that different. The goal is to develop a neural network that can classify their pirouettes.

This kind of tool would be useful in any physical activity. An athlete could train a neural network to learn any important movement. For example, a baseball pitcher’s pitch could be learned. The trained network could be used to compare the same movement at any other time to see if it has changed. A more sophisticated version, possibly including vision, might suggest how to fix problems or identify what has changed. This would be particularly valuable for rehabilitation.

Figure 7.1

Dancers doing pirouettes. The stages are from left to right.

Table 7.1 LPMS-B2: 9-Axis Inertial Measurement Unit (IMU).

Figure 7.2

LPMS-B2: 9-Axis Inertial Measurement Unit (IMU). The on/off button is highlighted on the right.

7.1.1 Inertial Measurement Unit

Our sensing means will be the LPMS-B2 IMU with its parameters shown in Table 7.1 that has Bluetooth. The range is sufficient to work in a ballet studio.

The IMU has many other outputs that we will not use. A close up of the IMU is shown in Figure 7.2.

We will first work out the details of the data acquisition. We will then build a deep learning algorithm to train the system and later to take data and classify the pirouette as being a pirouette done by a particular dancer. We’ll build up the data acquisition by first writing the MATLAB code to acquire the data. We will then create functions to display the data. We will then integrate it all into a GUI. Finally, we will create the deep learning classification system.

7.1.2 Physics

A pirouette is a complex multiflexible body problem. The pirouette is initiated by the dancer doing a pliè and then using his or her muscles to generate a torque about the spin axis and forces to get onto her pointe shoe and over her center of mass. Her muscles quickly stop the translational motion so that she can focus on balancing as she is turning. The equation of rotational motion is known as Euler’s equation and is

$$\displaystyle \begin{aligned} T = I\dot{\omega} + \omega^\times I \omega \end{aligned} $$

(7.1)

where ω is the angular rate and I is our inertia. T is our external torque. The external torque is due to a push off the floor and gravity. This is vector equation. The vectors are T and ω. I is a 3×3 matrix.

$$\displaystyle \begin{aligned} \begin{array}{rcl} T &=& \left[ \begin{array}{l} T_x\\ T_y\\ T_z \end{array} \right] \end{array} \end{aligned} $$

(7.2)

$$\displaystyle \begin{aligned} \begin{array}{rcl} \omega &=& \left[ \begin{array}{l} \omega_x\\ \omega_y\\ \omega_z \end{array} \right] \end{array} \end{aligned} $$

(7.3)

Each component is a value about a particular axis. For example, T _x is the torque about the x-axis attached to the dancer. Figure 7.3 shows the system. We will only be concerned with rotation assuming all translational motion is damped. If the dancer’s center of mass is not above the box of her pointe shoe, she will experience an overturning torque.

The dynamical model is three first-order couple differential equations. Angular rate ω is the state, that is, the quantity being differentiated. The equation says that the external torque (due to pushing off the floor or due to pointe shoe drag) is equal to the angular acceleration plus the Euler coupling term. This equation assumes that the body is rigid. For a dancer, it means she is rotating and no part is moving with respect to any other part. Now in a proper pirouette, this is never true if you are spotting! But let’s suppose you are one of those dancers who don’t spot. Let’s forget about the angular rate coupling term, which only matters if the angular rate is large. Let’s just look at the first two terms which are $T =I\dot {\omega }$. Expanded

$$\displaystyle \begin{aligned} \left[ \begin{array}{r} T_x\\ T_y\\ T_z \end{array} \right] = \left[ \begin{array}{rrr} I_{xx} & I_{xy} & I_{xz}\\ I_{xy} & I_{yy} & I_{yz}\\ I_{xz} & I_{yz} & I_{zz} \end{array} \right] \left[ \begin{array}{r} \dot{\omega}_x\\ \dot{\omega}_y\\ \dot{\omega}_z \end{array} \right] \end{aligned} $$

(7.4)

Let’s look at the equation for T _z. We just multiply the first row of the inertia matrix times the angular rate vector.

$$\displaystyle \begin{aligned} T_z = I_{xz} \dot{\omega}_x + I_{yz}\dot{\omega}_y + I_{zz}\dot{\omega}_z \end{aligned} $$

(7.5)

Figure 7.3

The center of mass of the dancer. External forces act at the center of mass. Rotations are about the center of mass.

This means a torque around the z-axis influences the angular rates about all 3 axes.

We can write out the required torque.

$$\displaystyle \begin{aligned} \left[ \begin{array}{r} T_x\\ T_y\\ T_z \end{array} \right] = \left[ \begin{array}{rrr} I_{xz}\\ I_{yz}\\ I_{zz} \end{array} \right] \dot{\omega}_z \end{aligned} $$

(7.6)

This is the perfect pirouette push off because it only creates rotation about the vertical axis, which is what we want in a pirouette. To this you need to add the forces needed to get on pointe with your center of mass over your pointe shoe tip.

While turning, the only significant external torque is due to friction between the pointe shoe tip and the floor. Friction resists both turning motion and translational motion. You don’t want a slight side force, perhaps due to a less than great partner, to cause you to slide.

Our IMU measures angular rates and linear accelerations. Angular rates are the quantities in Euler equations. However, since the IMU is not at the dancer’s center of mass, it will also measure angular accelerations along with the acceleration of the center of mass. We locate it at the dancer’s waist so it is not too far from the spin axis but it still sees a component.

$$\displaystyle \begin{aligned} a_\omega = r_{\mathrm{IMU}}^\times \dot\omega{} \end{aligned} $$

(7.7)

where r _IMU is the vector from the dancer’s center of mass to the IMU.

For a dancer doing a pirouette, Euler’s equation is not sufficient. A dancer can transfer momentum internally to stop a pirouette and needs a little jump to get on demi-pointe or pointe. To model this, we add additional terms.

$$\displaystyle \begin{aligned} \begin{array}{rcl} T &\displaystyle =&\displaystyle I\dot{\omega} + \omega^\times\left[I\omega + uI_i(\varOmega_i+\omega_z)+uI_h(\varOmega_h+\omega_z)\right]+ u\left(T_i+T_h\right) \end{array} \end{aligned} $$

(7.8)

$$\displaystyle \begin{aligned} \begin{array}{rcl} T_i &\displaystyle =&\displaystyle I_i\left(\dot{\varOmega}_i+ \dot{\omega}_z\right) \end{array} \end{aligned} $$

(7.9)

$$\displaystyle \begin{aligned} \begin{array}{rcl} T_h&\displaystyle =&\displaystyle I_h\left(\dot{\varOmega}_h+ \dot{\omega}_z\right) \end{array} \end{aligned} $$

(7.10)

$$\displaystyle \begin{aligned} \begin{array}{rcl} F&\displaystyle =&\displaystyle m\ddot{z} \end{array} \end{aligned} $$

(7.11)

where m is the mass, F is the vertical force, and z is the vertical direction. I _i is the internal inertia for control, and I _h is the head inertia. I includes both of these already. That is, I is the total body inertia that includes the internal “wheel,” body, and head. T _i is the internal torque. T _h is the head torque (for spotting). The internal torques, T _i, and T _h are between the body and the internal “wheel” or head. For example, T _h causes the head to move one way and the body the other. If you are standing, the torque you produce from your feet against the floor prevents your body from rotating. T is the external torque due to friction and the initial push off by the feet. The unit vector is

$$\displaystyle \begin{aligned} u = \left[ \begin{array}{l} 0\\ 0\\ 1 \end{array} \right]\end{aligned} $$

(7.12)

There are six equations in total. The first is a vector equation with three components, the second two are scalar equations. The vector equation is three equations, and each scalar equation is just one equation. We can use these to create a simulation of a dancer. The second component models all z-axis internal rotation, including spotting.

7.2 Data Acquisition

7.2.1 Problem

We want to get data from the Bluetooth IMU.

7.2.2 Solution

We will use the MATLAB bluetooth function. We’ll create a function to read data from the IMU.

Figure 7.4

The IMU is connected to the bluetooth device from a MacBook Pro via USB C and a ubiquitous Mac dongle. This is only for charging purposes. Once it is charged, you can use without the dongle.

7.2.3 How It Works

We will write an interface to the bluetooth device. First make sure the IMU is charged. Connect it to your computer as shown in Figure 7.4. Push the button on the back. This turns it on and off. The status is indicated by the LED. The IMU comes with support software from the vendor, but you will not need any of their software as MATLAB does all the hard work for you.

Let’s try commanding the IMU. Type btInfo = instrhwinfo('Bluetooth') and you should get the following:

 >> btInfo = instrhwinfo(’Bluetooth’)  btInfo =    HardwareInfo with properties:          RemoteNames: {’LPMSB2-4B31D6’}            RemoteIDs: {’btspp://00043E4B31D6’}      BluecoveVersion: ’BlueCove-2.1.1-SNAPSHOT’       JarFileVersion: ’Version 4.0’  Access to your hardware may be provided by a support package. Go to the Support Package Installer to learn more.

This shows that your IMU is discoverable. There is no support package available from the MathWorks. Now type b = Bluetooth(btInfo.RemoteIDs{1},1) (this can be slow). The number is the channel. The Bluetooth function requires the Instrument Control Toolbox for MATLAB.

 >> b = Bluetooth(btInfo.RemoteIDs{1},1)     Bluetooth Object : Bluetooth-btspp://00043E4B31D6:1     Communication Settings        RemoteName:         LPMSB2-4B31D6        RemoteID:           btspp://00043e4b31d6        Channel:            1        Terminator:         ’LF’     Communication State        Status:             closed        RecordStatus:       off     Read/Write State        TransferStatus:     idle        BytesAvailable:     0        ValuesReceived:     0        ValuesSent:         0

Note that the Communication State Status shows closed. We need to open the device by typing fopen(b). If you don’t have this device, just type

 >> btInfo = instrhwinfo(’Bluetooth’)  btInfo =    HardwareInfo with properties:          RemoteNames: []            RemoteIDs: []      BluecoveVersion: ’BlueCove-2.1.1-SNAPSHOT’       JarFileVersion: ’Version 4.0’  Access to your hardware may be provided by a support package. Go to the Support Package Installer to learn more.

This says it cannot recognize remote names or ids. You may need a support package for your device in this case.

Click connect and the device will open. Now type a= fscanf(b) and you will get a bunch of unprintable characters. We now have to write code to command the device. We will leave the device in streaming mode. The data unit format is shown in Table 7.2. Each packet is really 91 bytes long even though the table only shows 67 bytes. The 67 bytes are all the useful data.

We read the binary and put it into a data structure using DataFromIMU. typecast converts from bytes to float.

Table 7.2 Reply data.

DataFromIMU.m

25  function  d = DataFromIMU( a ) 27 d.packetStart   =  dec2hex (a(1)); 28 d.openMATIDLSB  =  dec2hex (a(2)); 29 d.openMATIDMSB  =  dec2hex (a(3)); 30 d.cmdNoLSB      =  dec2hex (a(4)); 31 d.cmdNoMSB      =  dec2hex (a(5)); 32 d.dataLenLSB    =  dec2hex (a(6)); 33 d.dataLenMSB    =  dec2hex (a(7)); 34 d.timeStamp     = BytesToFloat( a(8:11) ); 35 d.gyro          = [ BytesToFloat( a(12:15) );... 36                     BytesToFloat( a(16:19) );... 37                     BytesToFloat( a(20:23) )]; 38 d.accel         = [ BytesToFloat( a(24:27) );... 39                     BytesToFloat( a(28:31) );... 40                     BytesToFloat( a(32:35) )]; 41 d.quat          = [ BytesToFloat( a(48:51) );... 42                     BytesToFloat( a(52:55) );... 43                     BytesToFloat( a(56:59) );... 44                     BytesToFloat( a(60:63) )]; 45 d.msgEnd1       =  dec2hex (a(66)); 46 d.msgEnd2       =  dec2hex (a(67)); 49   %% DataFromIMU>BytesToFloat 50  function  r = BytesToFloat( x ) 52 r = typecast(uint8(x), ’single’);

We’ve wrapped all of this into the script BluetoothTest.m. We print out a few samples of the data to make sure our bytes are aligned correctly.

BluetoothTest.m

1   %% Script to read binary from the IMU 3   % Find available Bluetooth devices 4 btInfo = instrhwinfo( ’Bluetooth’) 6    % Display the information about the first device discovered 7 btInfo.RemoteNames(1) 8 btInfo.RemoteIDs(1) 10   % Construct a Bluetooth Channel object to the first Bluetooth device 11 b = Bluetooth(btInfo.RemoteIDs{1}, 1); 13   % Connect the Bluetooth Channel object to the specified remote device 14  fopen (b); 16   % Get a data structure 17  tic 18 t = 0; 19  for  k = 1:100 20   a   =  fread (b,91); 21   d   = DataFromIMU( a ); 22    fprintf ( ’%12.2f [%8.1e %8.1e %8.1e] [%8.1e %8.1e %8.1e] [%8.1f %8.1f %8.1f %8.1f]\n’,t,d.gyro,d.accel,d.quat); 23   t = t +  toc ; 24    tic 25  end

 When we run the script we get the following output.  >> BluetoothTest  btInfo =    HardwareInfo with properties:          RemoteNames: {’LPMSB2-4B31D6’}            RemoteIDs: {’btspp://00043E4B31D6’}      BluecoveVersion: ’BlueCove-2.1.1-SNAPSHOT’       JarFileVersion: ’Version 4.0’  Access to your hardware may be provided by a support package. Go to the Support Package Installer to learn more.  ans =    1x1 cell array      {’LPMSB2-4B31D6’}  ans =    1x1 cell array      {’btspp://00043E4B31D6’}  ans =    1x11 single row vector      1.0000    0.0014    0.0023   -0.0022    0.0019   -0.0105   -0.9896    0.9200   -0.0037    0.0144    0.3915  ans =    1x11 single row vector      2.0000   -0.0008    0.0023   -0.0016    0.0029   -0.0115   -0.9897    0.9200   -0.0037    0.0144    0.3915  ans =    1x11 single row vector      3.0000    0.0004    0.0023   -0.0025    0.0028   -0.0125   -0.9900    0.9200   -0.0037    0.0144    0.3915

The first number in each row vector is the sample, the next three are the angular rates from the gyro, the next three the accelerations, and the last four the quaternion. The acceleration is mostly in the -z direction which means that +z is in the button direction. Bluetooth, like all wireless connections, can be problematic. If you get this error

 Index exceeds the number of array elements (0).  Error in BluetoothTest (line 7)  btInfo.RemoteNames(1)

turn the IMU on and off. You might also have to restart MATLAB at times. This is because RemoteNames is empty, and this test is assuming it will not be. MATLAB then gets confused.

7.3 Orientation

7.3.1 Problem

We want to use quaternions to represent the orientation of our dancer in our deep learning system.

7.3.2 Solution

Implement basic quaternion operations. We need quaternion operations to process the quaternions from the IMU.

7.3.3 How It Works

Quaternions are the preferred mathematical representation of orientation. Propagating a quaternion requires fewer operations than propagating a transformation matrix and avoids singularities that occur with Euler angles. A quaternion has four elements, which corresponds to a unit vector a and angle of rotation ϕ about that vector. The first element is termed the “scalar component” s, and the next three elements are the “vector” components v. This notation is shown as follows [21]:

$$\displaystyle \begin{aligned} q = \left[ \begin{array}{c} q_{0}\\ q_{1}\\ q_{2}\\ q_{3}\\ \end{array} \right] = \left[ \begin{array}{c} s\\ v_{1}\\ v_{2}\\ v_{3}\\ \end{array} \right] = \left[ \begin{array}{c} \cos\frac{\phi}{2}\\ a_{1}\sin\frac{\phi}{2}\\ a_{2}\sin\frac{\phi}{2}\\ a_{3}\sin\frac{\phi}{2}\\ \end{array} \right] \end{aligned} $$

(7.13)

The “unit” quaternion which represents zero rotation from the initial coordinate frame has a unit scalar component and zero vector components. This is the same convention used on the Space Shuttle, although other conventions are possible.

$$\displaystyle \begin{aligned} q_{0}= \left[ \begin{array}{c} 1\\ 0\\ 0\\ 0\\ \end{array} \right] \end{aligned} $$

(7.14)

In order to transform a vector from one coordinate frame a to another b using a quaternion q _ab, the operation is

$$\displaystyle \begin{aligned} u_{b}=q_{ab}^{T}u_{a}q_{ab} \end{aligned} $$

(7.15)

using quaternion multiplication with the vectors defined as quaternions with a scalar part equal to zero, or

$$\displaystyle \begin{aligned} x_{a} = \left[ \begin{array}{l} 0\\ x_{a}(1)\\ x_{a}(2)\\ x_{a}(3)\\ \end{array} \right] \end{aligned} $$

(7.16)

For example, the quaternion

$$\displaystyle \begin{aligned} \left[ \begin{array}{l} 0.7071\\ 0.7071\\ 0.0\\ 0.0 \end{array} \right] \end{aligned} $$

(7.17)

represents a pure rotation about the x-axis. The first element is 0.7071 and equals the $\cos {}(90^{\circ }/2)$. We cannot tell the direction of rotation from the first element. The second element is the 1 component of the unit vector, which in this case is

$$\displaystyle \begin{aligned} \left[ \begin{array}{l} 1.0\\ 0.0\\ 0.0 \end{array} \right] \end{aligned} $$

(7.18)

times the argument $\sin {}(90^\circ /2)$. Since the sign is positive, the rotation must be a positive 90^∘ rotation.

We only need one routine that converts the quaternion, which comes from the IMU, into a transformation matrix for visualization. We do this because multiplying a 3× n array of vectors for the vertices of our 3D model by a matrix is much faster than transforming each vector with a quaternion.

QuaternionToMatrix.m

1       2*(q(2)*q(4)+q(1)*q(3));... 2       2*(q(2)*q(3)+q(1)*q(4)),... 3       q(1)^2-q(2)^2+q(3)^2-q(4)^2,... 4       2*(q(3)*q(4)-q(1)*q(2));... 5       2*(q(2)*q(4)-q(1)*q(3)),... 6       2*(q(3)*q(4)+q(1)*q(2)),... 7       q(1)^2-q(2)^2-q(3)^2+q(4)^2];

Note that the diagonal terms have the same form. The off-diagonal terms also all have the same form.

7.4 Dancer Simulation

7.4.1 Problem

We want to simulate a dancer for readers who don’t have access to the hardware.

7.4.2 Solution

We will write a right-hand side for the dancer based on the preceding equations and write a simulation with a control system.

7.4.3 How It Works

The right-hand side implements the dancer model. It includes an internal control “wheel” and a degree of freedom for the head movement. The default data structure is returned if you call it without arguments.

RHSDancer.m

33   %% RHSDANCER Implements dancer dynamics 34   % This is a model of dancer with one degree of translational freedom 35   % and 5 degrees of rotational freedom including the head and an internal 36   % rotational degree of freedom. 37   %% Form: 38   %   xDot = RHSDancer( x, ~, d ) 39   %% Inputs 40   %   x       (11,1)    State vector [r;v;q;w;wHDot;wIDot] 41   %   t       (1,1)   Time (unused) (s) 42   %   d       (1,1)   Data structure for the simulation 43   %                   .torque   (3,1) External torque (Nm) 44   %                   .force    (1,1) External force (N) 45   %                   .inertia  (3,3) Body inertia (kg-m^2) 46   %                   .inertiaH (1,1) Head inertia (kg-m^2) 47   %                   .inertiaI (1,1) Inner inertia (kg-m^2) 48   %                   .mass     (1,1) Dancer mass (kg) 49   % 50   %% Outputs 51   %   xDot    (11,1)    d[r;v;q;w;wHDot;wIDot]/dt 53  function  xDot = RHSDancer( ~, x, d ) 55   % Default data structure 56  if (  nargin  < 1 ) 57      % Based on a 0.15 m radius, 1.4 m long cylinders 58   inertia =  diag ([8.4479    8.4479    0.5625]); 59   xDot    = struct( ’torque’,[0;0;0], ’force’,0, ’inertia’,inertia,... 60    ’mass’,50, ’inertiaI’,0.0033, ’inertiaH’,0.0292, ’torqueH’,0, ’torqueI’,0); 61    return 62  end

The remainder mechanizes the equations given earlier. We add an additional equation for the integral of the z-axis rate. This makes the control system easier to write. We also include the gravitational acceleration in the force equation.

63   % Use local variables 64 v     = x(2); 65 q     = x(3:6); 66 w     = x(7:9); 67 wI    = x(10); 68 wH    = x(11); 70   % Unit vector 71 u     = [0;0;1]; 73   % Gravity 74 g     = 9.806; 76   % Attitude kinematics (not mentioned in the text) 77 qDot   = QIToBDot( q, w ); 79   % Rotational dynamics Equation 7.6 80 wDot   = d.inertia\(d.torque - Skew(w)*(d.inertia*w + d.inertiaI*(wI + w(3))... 81       + d.inertiaH*(wH + w(3))) - u*(d.torqueI + d.torqueH)); 82 wHDot = d.torqueH/d.inertiaH - wDot(3); 83 wIDot = d.torqueI/d.inertiaI - wDot(3); 85   % Translational dynamics 86 vDot  = d.force/d.mass - g; 88   % Assemble the state vector 89 xDot   = [v; vDot; qDot; wDot; wHDot; wIDot; w(3)];

The simulation setup gets default parameters from RHSDancer.

1 d       = RHSDancer; 2 n       = 800; 3 dT      = 0.01; 4 xP      =  zeros (16,n); 5 x       =  zeros (12,1); 6 x(3)    = 1; 7 g       = 9.806; 8 dancer  =  ’Robot_1’;

It then sets up the control system. We use a proportional derivative controller for z position and a rate damper to stop the pirouette. The position control is done by the foot muscles. The rate damping is our internal damper wheel.

13   % Control system for 2 pirouettes in 6 seconds 14 tPirouette  = 6; 15 zPointe     = 6*0.0254; 16 tPointe     = 0.1; 17 kP          = tPointe/dT; 18 omega       = 4* pi /tPirouette; 19 torquePulse = d.inertia(3,3)*omega/tPointe; 20 tFriction   = 0.1; 21 a           = 2*zPointe/tPointe^2 + g; 22 kForce      = 1000; 23 tau         = 0.5; 24 thetaStop   = 4* pi  -  pi /4; 25 kTorque     = 200; 26 state       =  zeros (10,n);

The simulation loop calls the right-hand side and the control system. We call RHSDancer.m to get the linear acceleration.

1   %% Simulate 2  for  k = 1:n 3   d.torqueH = 0; 4   d.torqueI = 0; 6     % Get the data for use in the neural network 7   xDot = RHSDancer(0,x,d); 9   state(:,k) = [x(7:9);0;0;xDot(2);x(3:6)]; 11     % Control 12    if ( k < kP ) 13     d.force  = d.mass*a; 14     d.torque = [0;0;torquePulse]; 15    else 16     d.force  = kForce*(zPointe-x(1) -x(2)/tau)+ d.mass*g; 17     d.torque = [0;0;-tFriction]; 18    end 20    if ( x(12) > thetaStop ) 21     d.torqueI = kTorque*x(9); 22    end 24   xP(:,k)   = [x;d.force;d.torque(3);d.torqueH;d.torqueI]; 25   x         = RungeKutta(@RHSDancer,0,x,dT,d); 26  end

The control system includes a torque and force pulse to get the pirouette going.

1     % Control 2    if ( k < kP ) 3     d.force  = d.mass*a; 4     d.torque = [0;0;torquePulse]; 5    else 6     d.force  = kForce*(zPointe-x(1) -x(2)/tau)+ d.mass*g; 7     d.torque = [0;0;-tFriction]; 8    end

The remainder of the script plots the results and outputs the data, which would have come from the IMU, into a file.

Simulation results for a double pirouette are shown in Figure 7.5. We stop the turn at 6.5 seconds, hence the pulse.

You can create different dancers by varying the mass properties and the control parameters.

 zPointe     = 6*0.0254;  tPointe     = 0.1;  tFriction   = 0.1;  kForce      = 1000;  tau         = 0.5;  kTorque     = 200;

We didn’t implement spotting control (looking at the audience as much as possible during the turn). It would rotate the head so that it faces forward whenever the head was within 90 degrees or so of front. We’d need to add a head angle for that purpose to the right-hand side, much like we added the z-axis angle.

Figure 7.5

Simulation of a double pirouette.

7.5 Real-Time Plotting

7.5.1 Problem

We want to display data from the IMU in real time. This will allow us to monitor the pirouettes.

7.5.2 Solution

Use plot with drawnow to implement multiple figures of plots.

7.5.3 How It Works

The main function is a switch statement with two cases. The function also has a built-in demo. The first case, initialize, initializes the plot figures. It stores everything in a data structure that is returned on each function call. This is one way for a function to have a memory. We return the data structure from each subfunction.

GUIPlots.m

27 switch(  lower (action) ) 28   case  ’initialize’ 29     g = Initialize( g ); 31   case  ’update’ 32     g = Update( g, y, t ); 34  end

The first case, initialize, initializes the figure window.

35   %% GUIPlots>Initialize 36  function  g = Initialize( g ) 38 lY =  length (g.yLabel); 40   % Create tLim if it does not exist 41  if ( ~isfield(g,  ’tLim’ ) ) 42   g.tLim = [0 1]; 43  end 45 g.tWidth = g.tLim(2) - g.tLim(1); 47   % Create yLim if it does not exist 48  if ( ~isfield( g,  ’yLim’ ) ) 49   g.yLim = [- ones (lY,1),  ones (lY,1)]; 50  end 52   % Create the plots 53 lP =  length (g.yLabel); 54 y  = g.pos(2);   % The starting y position 55  for  k = 1:lP 56   g.h(k) =  subplot (lP,1,k); 57    set (g.h(k), ’position’,[g.pos(1) y g.pos(3) g.pos(4)]); 58   y = y - 1.4*g.pos(4); 59   g.hPlot(k) =  plot (0,0); 60   g.hAxes(k) =  gca ; 61   g.yWidth(k) = (g.yLim(k,2) - g.yLim(k,1))/2; 62    set (g.hAxes(k), ’nextplot’, ’add’, ’xlim’,g.tLim); 63    ylabel ( char(g.yLabel{k}) ) 64    grid  on 65  end 66  xlabel ( g.tLabel );

The second case, update, updates the data displayed in the plot. It leaves the existing figures, subplots, and labels in place and just updates the plots of the line segments with new data. It can change the size of the axes as needed. The function adds a line segment for each new data point. This way no storage is needed external to the plot. It reads xdata and ydata and appends the new data to those arrays.

67  function  g = Update( g, y, t ) 69   % See if the time limits have been exceeded 70  if ( t > g.tLim(2) ) 71   g.tLim(2)  = g.tWidth + g.tLim(2); 72   updateAxes = true; 73  else 74   updateAxes = false; 75  end 77 lP =  length (g.yLabel); 78  for  k = 1:lP 79    subplot (g.h(k)); 80   yD =  get (g.hPlot(k), ’ydata’); 81   xD =  get (g.hPlot(k), ’xdata’); 82    if ( updateAxes ) 83      set (  gca ,  ’xLim’, g.tLim ); 84      set ( g.hPlot(k),  ’xdata’,[xD t], ’ydata’,[yD y(k)]); 85    else 86      set ( g.hPlot(k),  ’xdata’,[xD t], ’ydata’,[yD y(k)] ); 87    end 89  end

The built-in demo plots six numbers. It updates the axes in time once. It sets up a figure window with six plots. You need to create the figure and save the figure handle before calling GUIPlots.

 g.hFig  = NewFig(’State’);

The pause in the demo just slows down plotting so that you can see the updates. The height (the last number in g.pos) is the height of each plot. If you happen to set the locations of the plots out of the figure window, you will get a MATLAB error. g.tLim gives the initial time limits in second. The upper limit will expand as data is entered.

2  function  Demo 4 g.yLabel = { ’x’  ’y’  ’z’  ’x_1’  ’y_1’  ’z_1’}; 5 g.tLabel =  ’Time (sec)’; 6 g.tLim   = [0 100]; 7 g.pos    = [0.100    0.88    0.8    0.10]; 8 g.width  = 1; 9 g.color  =  ’b’; 11 g.hFig  = NewFig( ’State’); 12  set (g.hFig,  ’NumberTitle’, ’off’ ); 14 g        = GUIPlots(  ’initialize’, [], [], g ); 16  for  k = 1:200 17   y = 0.1*[ cos ((k/100))-0.05; sin (k/100)]; 18   g = GUIPlots(  ’update’, [y;y.^2;2*y], k, g ); 19    pause (0.1) 20  end 22 g        = GUIPlots(  ’initialize’, [], [], g ); 24  for  k = 1:200 25   y = 0.1*[ cos ((k/100))-0.05; sin (k/100)]; 26   g = GUIPlots(  ’update’, [y;y.^2;2*y], k, g ); 27    pause (0.1) 28  end

Figure 7.6 shows the real-time plots at the end of the demo.

7.6 Quaternion Display

7.6.1 Problem

We want to display the dancer’s orientation in real time.

Figure 7.6

Real-time plots.

7.6.2 Solution

Use patch to draw an OBJ model in a three-dimensional plot. The figure is easier to understand than the four quaternion elements. Our solution can handle 3-axis rotation although typically we will only see single-axis rotation.

7.6.3 How It Works

We start with our Ballerina.obj file. It only has vertices and faces. A 3D drawing consists of a set of vertices. Each vertex is a point in space. The vertices are organized into faces. Each face is a triangle. Triangles are used for 3D drawings because they always form a plane. 3D processing hardware is designed to work with triangles, so this also gives the fastest results. The obj files for our software can only contain triangles. Each face can have only three vertices. Generally, obj files can have any size polynomials, that is, faces with more than three points. Most sources of obj files can provide tessellation services to convert polygons with more than three vertices into triangles. LoadOBJ.m will not draw models with anything other than triangular faces.

The main part of the function uses a case statement to handle the three actions. The first action just returns the defaults, which is the name of the default obj file. The second reads in the file and initializes the patches. The third updates the patches. patch is the MATLAB name for a set of triangles. The function can be passed a figure handle. A figure handle tells it into which figure the 3D model should be drawn. This allows it to be used as part of a GUI, as will be shown in the next section.

QuaternionVisualization.m

19  function  m = QuaternionVisualization( action, x, f ) 21 persistent p 23   % Demo 24  if (  nargin  < 1 ) 25   Demo 26    return 27  end 29 switch(  lower (action) ) 30     case  ’defaults’ 31         m = Defaults; 33     case  ’initialize’ 34          if (  nargin  < 2 ) 35             d   = Defaults; 36          else 37             d   = x; 38          end 40          if (  nargin  < 3 ) 41           f = []; 42          end 44         p = Initialize( d, f ); 46     case  ’update’ 47          if (  nargout  == 1 ) 48             m = Update( p, x ); 49          else 50             Update( p, x ); 51          end 52  end

Initialize loads the obj file. It creates a figure and saves the object data structure. It sets shading to interpolated and lighting to Gouraud. Gouraud is a type of lighting model named after its inventor. It then creates the patches and sets up the axis system. We save handles to all the patches for updating later. We also place a light.

53  function  p = Initialize( file, f ) 55  if (  isempty (f) ) 56   p.fig = NewFigure(  ’Quaternion’ ); 57  else 58   p.fig = f; 59  end 61 g     = LoadOBJ( file ); 62 p.g   = g; 64  shading  interp 65 lighting gouraud 67 c = [0.3 0.3 0.3]; 69  for  k = 1: length (g.component) 70   p.model(k)   =  patch ( ’vertices’, g.component(k).v,  ’faces’, g.component(k).f,  ’facecolor’,c, ’edgecolor’,c, ’ambient’,1, ’edgealpha’,0 ); 71  end 73  xlabel ( ’x’); 74  ylabel ( ’y’); 75  zlabel ( ’z’); 77  grid 78 rotate3d on 79  set ( gca , ’DataAspectRatio’,[1 1 1], ’DataAspectRatioMode’, ’manual’) 81 light( ’position’,10*[1 1 1]) 83  view ([1 1 1])

In Update we convert the quaternion to a matrix, because it is faster to matrix multiply all the vertices with one matrix multiplication. The vertices are n by 3 so we transpose before the matrix multiplication. We use the patch handles to update the vertices. The two options at the end are to create movie frames or just update the drawing.

84  function  m = Update( p, q ) 86 s = QuaternionToMatrix( q ); 88  for  k = 1: length (p.model) 89   v = (s*p.g.component(k).v’)’; 90    set (p.model(k), ’vertices’,v); 91  end 93  if (  nargout  > 0 ) 94         m =  getframe ; 95  else 96          drawnow ; 97  end

This is the built-in demo. We vary the 1 and 4 elements of the quaternion to get rotation about the z-axis.

109  function  Demo 111 QuaternionVisualization(  ’initialize’,  ’Ballerina.obj’ );

Figure 7.7

Dancer orientation. The obj file is by the artist loft_22 and is available from TurboSquid.

Figure 7.7 shows two orientations of the dancer during the demo. The demo produces an animation of the dancer rotating about the z-axis.

The rotation is slow because of the number of vertices. The figure is not articulated so the entire figure is rotated as a rigid body. MATLAB doesn’t make it easy to texture map so we don’t bother. In any case, the purpose of this function is just to show orientation so it doesn’t matter.

7.7 Data Acquisition GUI

7.7.1 Problem

Build a data acquisition GUI to display the real-time data and output it into training sets.

7.7.2 Solution

Integrate all the preceding recipes into a GUI.

7.7.3 How It Works

We aren’t going to use MATLAB’s Guide to build our GUI. We will hand code it, which will give you a better idea of how a GUI really works.

We will use nested functions for the GUI. The inner functions have access to all variables in the outer functions. This also makes using callbacks easy as shown in the following code snippet.

 function DancerGUI( file )  function DrawGUI(h)   uicontrol( h.fig,’callback’,@SetValue);     function SetValue(hObject, ~, ~ )     % do something    end  end  end

A callback is a function called by a uicontrol when the user interacts with the control. When you first open the GUI, it will look for the bluetooth device. This can take a while.

Everything in DrawGUI has access to variables in DancerGUI. The GUI is shown in Figure 7.8. The 3D orientation display is in the upper left corner. Real-time plots are on the right. Buttons are on the lower left, and the movie window is on the right.

The upper left picture shows the dancer’s orientation. The plots on the right show angular rates and accelerations from the IMU. From top to bottom of the buttons

• 1. Turn the 3D on/off. The default model is big, so unless you add your own model with fewer vertices, it should be set to off.

• 2. The text box to its right is the name of the file. The GUI will add a number to the right of the name for each run.

• 3. Save saves the current data to a file.

• 4. Calibrate sets the default orientation and sets the gyro rates and accelerations to whatever it is reading when you hit the button. The dancer should be still when you hit calibrate. It will automatically compute the gravitational acceleration and subtract it during the test.

• 5. Quit closes the GUI.

• 6. Clear data clears out all the internal data storage.

• 7. Start/Stop starts and stops the GUI.

The remaining three lines display the time, the angular rate vector, and the acceleration vector as numbers. This is the same data that is plotted.

The first part creates the figure and draws the GUI. It initializes all the fields for GUIPlots. It reads in a default picture for the movie window as a placeholder.

DancerGUI.m

16  function  DancerGUI( file ) 18   % Demo 19  if (  nargin  < 1 ) 20   DancerGUI( ’Ballerina.obj’); 21    return 22  end 24   % Storage of data need by the deep learning system 25 kStore      = 1; 26 accelStore  =  zeros (3,1000); 27 gyroStore   =  zeros (3,1000); 28 quatStore   =  zeros (4,1000); 29 timeStore   =  zeros (1,1000); 30 time        = 0; 31 on3D        = false; 32 quitNow     = false; 34 sZ =  get (0, ’ScreenSize’) + [99 99 -200 -200]; 36 h.fig =  figure ( ’name’, ’Dancer Data Acquisition’, ’position’,sZ, ’units’, ’pixels’,... 37    ’NumberTitle’, ’off’, ’tag’, ’DancerGUI’, ’color’,[0.9 0.9 0.9]); 39   % Plot display 40 gPlot.yLabel  = { ’\omega_x’  ’\omega_y’  ’\omega_z’  ’a_x’  ’a_y’  ’a_z’}; 41 gPlot.tLabel  =  ’Time (sec)’; 42 gPlot.tLim    = [0 100]; 43 gPlot.pos     = [0.45    0.88    0.46    0.1]; 44 gPlot.color   =  ’b’; 45 gPlot.width   = 1; 47   % Calibration 48 q0            = [1;0;0;0]; 49 a0            = [0;0;0]; 51 dIMU.accel    = a0; 52 dIMU.quat     = q0; 54   % Initialize the GUI 55 DrawGUI;

Figure 7.8

Data acquisition GUI.

The notation

1  ’\omega_x’

is latex format. This will generate ω _x.

The next part tries to find Bluetooth. It first sees if Bluetooth is available at all. It then enumerates all Bluetooth devices. It looks through the list to find our IMU.

2  if ( ~ isempty (btInfo.RemoteIDs) ) 3      % Display the information about the first device discovered 4   btInfo.RemoteNames(1) 5   btInfo.RemoteIDs(1) 6    for  iB =  length (btInfo.RemoteIDs) 7      if (  strcmp (btInfo.RemoteNames(iB), ’LPMSB2-4B31D6’) ) 8        break ; 9      end 10    end 11   b       = Bluetooth(btInfo.RemoteIDs{iB}, 1); 12    fopen (b);   % No output allowed for some reason 13   noIMU   = false; 14   a       =  fread (b,91); 15   dIMU    = DataFromIMU( a ); 16  else 17   warndlg( ’The IMU is not available.’,  ’Hardware Configuration’) 18   noIMU   = true; 19  end

The following is the run loop. If no IMU is present, it synthesizes data. If the IMU is found, the GUI reads data from the IMU in 91 byte chunks. The uiwait is to wait until the user hits the start button. When used for testing, the IMU should be on the dancer. The dancer should remain still when the start button is pushed. It will then calibrate the IMU. Calibration fixes the quaternion reference and removes the gravitational acceleration. You can also hit the calibration button at any time.

20  % Wait for user input 21 uiwait; 22  % The run loop 23 time  = 0; 24 tic 25 while(1) 26   if( noIMU ) 27     omegaZ      = 2*pi; 28     dT          = toc; 29     time        = time + dT; 30     tic 31     a           = omegaZ*time; 32     q           = [cos(a);0;0;sin(a)]; 33     accel       = [0;0;sin(a)]; 34     omega       = [0;0;omegaZ]; 35   else 36      % Query the bluetooth device 37     a       = fread(b,91); 38     pause(0.1);  % needed so not to overload the bluetooth device 40     dT      = toc; 41     time    = time + dT; 42     tic 44      % Get a data structure 45     if( length(a) > 1 ) 46       dIMU    = DataFromIMU( a ); 47     end 48     accel   = dIMU.accel - a0; 49     omega   = dIMU.gyro; 50     q       = QuaternionMultiplication(q0,dIMU.quat); 52     timeStore(1,kStore)   = time; 53     accelStore(:,kStore)  = accel; 54     gyroStore(:,kStore)   = omega; 55     quatStore(:,kStore)   = q; 56     kStore = kStore + 1; 57   end

 

58       dIMU    = DataFromIMU( a ); 59      end 60     accel   = dIMU.accel - a0; 61     omega   = dIMU.gyro;

This code closes the GUI and displays the IMU data.

62    if ( quitNow ) 63      close ( h.fig ) 64      return 65    else 66      if ( on3D ) 67       QuaternionVisualization(  ’update’, q ); 68      end 69      set (h. text (1), ’string’, sprintf ( ’[%5.2f;%5.2f;%5.2f] m/s^2’,accel)); 70      set (h. text (2), ’string’, sprintf ( ’[%5.2f;%5.2f;%5.2f] rad/s’,omega)); 71      set (h. text (3), ’string’,datestr(now)); 72     gPlot = GUIPlots(  ’update’, [omega;accel], time, gPlot ); 73    end 74  end

The drawing code uses uicontrol to create all the buttons. GUIPlots and QuaternionVisualization are also initialized. The uicontrol that require an action have callbacks.

75    if ( quitNow ) 76      close ( h.fig ) 77      return 78    else 79      if ( on3D ) 80       QuaternionVisualization(  ’update’, q ); 81      end 82      set (h. text (1), ’string’, sprintf ( ’[%5.2f;%5.2f;%5.2f] m/s^2’,accel)); 83      set (h. text (2), ’string’, sprintf ( ’[%5.2f;%5.2f;%5.2f] rad/s’,omega)); 84      set (h. text (3), ’string’,datestr(now)); 85     gPlot = GUIPlots(  ’update’, [omega;accel], time, gPlot ); 86    end 87  end 89   %% DancerGUI>DrawButtons 90  function  DrawGUI 92     % Plots 93   gPlot = GUIPlots(  ’initialize’, [], [], gPlot ); 95     % Quaternion display 96    subplot ( ’position’,[0.05 0.5 0.4 0.4], ’DataAspectRatio’,[1 1 1], ’PlotBoxAspectRatio’,[1 1 1] ); 97   QuaternionVisualization(  ’initialize’, file, h.fig ); 99     % Buttons 100   f   = { ’Acceleration’,  ’Angular Rates’  ’Time’}; 101   n   =  length (f); 102   p   =  get (h.fig, ’position’); 103   dY  = p(4)/20; 104   yH  = p(4)/21; 105   y   = 0.5; 106   x   = 0.15; 107   wX  = p(3)/6; 109     % Create pushbuttons and defaults 110    for  k = 1:n 111     h.pushbutton(k) =  uicontrol ( h.fig, ’style’, ’text’, ’string’,f{k}, ’position’,[x    y wX yH]); 112     h. text (k)       =  uicontrol ( h.fig, ’style’, ’text’, ’string’, ’’,   ’position’,[x+wX y 2*wX yH]); 113     y               = y + dY; 114    end

uicontrol takes parameter pairs, except for the first argument that can be a figure handle. There are lot of parameter pairs. The easiest way to explore them is to type

 h = uicontrol;  get(h)

All types of uicontrol that handle user interaction have “callbacks” that are functions that do something when the button is pushed or menu item is selected. We have five uicontrol with callbacks. The first uses uiwait and uiresume to start and stop data collection.

3     % Start/Stop button callback 4    function  StartStop(hObject, ~, ~ ) 5      if ( hObject.Value ) 6       uiresume; 7      else 8       SaveFile; 9       uiwait 10      end 11    end

The second uses questdlg to ask if you want to save the data that has been stored in the GUI. This produces the modal dialog shown in Figure 7.9.

12     % Quit button callback 13    function  Quit(~, ~, ~ ) 14     button = questdlg( ’Save Data?’, ’Exit Dialog’, ’Yes’, ’No’, ’No’); 15     switch button 16       case  ’Yes’ 17           % Save data 18       case  ’No’ 19      end 20     quitNow = true; 21     uiresume 22    end

Figure 7.9

Modal dialog.

The third, Clear, clears the data storage arrays. It resets the quaternion to a unit quaternion.

23     % Clear button callback 24    function  Clear(~, ~, ~ ) 25     kStore      = 1; 26     accelStore  =  zeros (3,1000); 27     gyroStore   =  zeros (3,1000); 28     quatStore   =  zeros (4,1000); 29     timeStore   =  zeros (1,1000); 30     time        = 0; 31    end

The fourth, calibrate, runs the calibration procedure.

32     % Calibrate button callback 33    function  Calibrate(~, ~, ~ ) 34     a       =  fread (b,91); 35     dIMU    = DataFromIMU( a ); 36     a0      = dIMU.accel; 37     q0      = dIMU.quat; 38     QuaternionVisualization(  ’update’, q0 ) 39    end

The fifth, SaveFile, saves the recorded data into a mat file for use by the deep learning algorithm.

40     % Save button call back 41    function  SaveFile(~,~,~) 42      cd  TestData 43     fileName =  get (h.matFile, ’string’); 44     s =  dir ; 45     n =  length (s); 46     fNames =  cell (1,n-2); 47      for  kF = 3:n 48       fNames{kF-2} = s(kF).name(1: end -4); 49      end 50     j = contains(fNames,fileName); 51     m = 0; 52      if ( ~ isempty (j) ) 53        for  kF = 1: length (j) 54          if ( j(kF)) 55           f = fNames{kF}; 56           i = strfind(f, ’_’); 57           m = str2double(f(i+1: end )); 58          end 59        end 60      end

We make it easier for the user to save files by reading the directory and adding a number to the end of the dancer filename that is one greater than the last filename number.

7.8 Making the IMU Belt

7.8.1 Problem

We need to attach the IMU to our dancer.

7.8.2 Solution

We use the arm strap that is available from the manufacturer. We buy an elastic belt and make one that fits around the dancer’s waist.

7.8.3 How It Works

Yes, software engineers need to sew. Figure 7.10 shows the process. The two products used to make the data acquisition belt are

• 1. LPMS-B2 Holder (available from Life Performance Research)

• 2. Men’s No Show Elastic Stretch Belt Invisible Casual Web Belt Quick Release Flat Plastic Buckle (available from Amazon)

Remove the holder from the LPMS-B2 Holder. Cut the belt at the buckle and slide the holder onto the belt. Sew the belt at the buckle.

The sensor on a dancer is shown in Figure 7.11. We had the dancer stand near the laptop during startup. We didn’t have any range problems during the experiments. We didn’t try it with across the floor movement as one would have during grande allegro.

Figure 7.10

Elastic belt manufacturing. We use the two items on the left to make the one on the right.

Figure 7.11

Dancer with the sensor belt. The blue light means it is collecting data.

7.9 Testing the System

7.9.1 Problem

We want to test the data acquisition system. This will find any problems with the data acquisition process.

7.9.2 Solution

Have a dancer do changements, which are small jumps changing the foot position on landing.

7.9.3 How It Works

The dancer puts on the sensor belt, we push the calibrate button, then she does a series of changements. She stands about 2 m from your computer to make acquisition easier. The dancer will do small jumps, known as changements. A changement is a small jump where the feet change positions starting from fifth position. If the right foot is in fifth position front at the start, it is in the back at the finish. Photos are shown in Figure 7.12.

The time scale is a bit long. You can see that the calibration does not lead us to a natural orientation in the axis system in the GUI. It doesn’t matter from a data collection point of view but is an improvement we should make in the future. The changement is shown in Figure 7.13. The dancer is still at the beginning and end.

The interface to the bluetooth device doesn’t do any checking or stream control. Some bluetooth data collection errors occur from time to time. Typically, they happen after 40 seconds of data collection.

Figure 7.12

Dancer doing a changement. Notice the feet when she is preparing to jump. The second image shows her feet halfway through the jump.

Figure 7.13

Data collected during the changement.

 index exceeds the number of array elements (0).  Error in instrhwinfo>bluetoothCombinedDevices (line 976)          uniqueBTName = allBTName(uniqueRowOrder);  Error in instrhwinfo (line 206)                      tempOut = bluetoothCombinedDevices(tempOut);  Error in DataAcquisition (line 13)  btInfo  = instrhwinfo(’Bluetooth’);

If this happens, turn the device on and off. Restart MATLAB if that doesn’t work.

Another bluetooth error is

 ans =    1x1 cell array      {’LPMSB2-4B31D6’}  ans =    1x1 cell array      {’btspp://00043E4B31D6’}  Error using Bluetooth (line 104)  Cannot Create: Java exception occurred:  java.lang.NullPointerException          at com.mathworks.toolbox.instrument.BluetoothDiscovery.searchDevice(BluetoothDiscovery.java:395)          at com.mathworks.toolbox.instrument.BluetoothDiscovery.discoverServices(BluetoothDiscovery.java:425)          at com.mathworks.toolbox.instrument.BluetoothDiscovery.hardwareInfo(BluetoothDiscovery.java:343)          at com.mathworks.toolbox.instrument.Bluetooth.<init>(Bluetooth.java:205).

This is a MATLAB error and requires restarting MATLAB. It doesn’t happen very often. We ran the entire data collection with four dancers doing ten pirouettes each without ever experiencing the problem.

7.10 Classifying the Pirouette

7.10.1 Problem

We want to classify the pirouettes of our four dancers.

7.10.2 Solution

Create an LSTM that classifies pirouettes according to dancer. The four labels are the dancer names.

Figure 7.14

A pirouette. Angular rate and linear acceleration are shown.

7.10.3 How It Works

The script takes one file and displays it. Figure 7.14 shows a double pirouette. A turn takes only a few seconds.

1 dancer = { ’Ryoko’  ’Shaye’,  ’Emily’,  ’Matanya’}; 3   %% Show one dancer’s data 4  cd  TestData 5 s   =  load ( ’Ryoko_10.mat’); 6 yL  = { ’\omega_x’  ’\omega_y’  ’\omega_z’  ’a_x’  ’a_y’  ’a_z’}; 7 PlotSet(s.time,s.state(1:6,:), ’x label’, ’Time (s)’, ’y label’,yL, ’figure title’,dancer{1});

We load in the data and limit the range to 6 seconds. Sometimes the IMU would run longer due to human error. We also remove sets that are bad.

DancerNN.m

1   %% Load in and process the data 2 n = 40; 3   % Get the data and remove bad data sets 4 i = 0; 5  for  k = 1: length (dancer) 6    for  j = 1:10 7     s   =  load ( sprintf ( ’%s_%d.mat’,dancer{k},j)); 8     cS  =  size (s.state,2); 9      if ( cS > 7 ) 10       i       = i + 1; 11       d{i,1}  = s.state;   %#ok<*SAGROW> 12       t{i,1}  = s.time; 13       c(i,1)  = k; 14      end 15    end 16  end 18  fprintf ( ’%d remaining data sets out of %d total.\n’,i,n) 20  for  k = 1:4 21   j =  length ( find (c==k)); 22    fprintf ( ’%7s data sets %d\n’,dancer{k},j) 23  end 25 n = i; 27  cd  .. 29   % Limit the range to 6 seconds 30 tRange = 6; 31  for  i = 1:n 32   j =  find (t{i} - t{i,1} > tRange ); 33    if ( ~ isempty (j) ) 34     d{i}(:,j(1)+1: end )= []; 35    end 36  end

We then train the neural network. We use a bidirectional LSTM to classify the sequences. There are ten features, four quaternion measurements, three rate gyro, and three accelerometer. The four quaternion numbers are coupled through the relationship

$$\displaystyle \begin{aligned} 1 = q_1^2 + q_2^2 + q_3^2 + q_4^2 \end{aligned} $$

(7.19)

However, this should not impact the learning accuracy aside from slowing down the learning.

We then load in and process the data. Some data sets didn’t have any data and need to be removed. We also limit the range to 6 seconds since sometimes the data collection did not stop after the pirouette ended.

1   %% Load in and process the data 2 n = 40; 3   % Get the data and remove bad data sets 4 i = 0; 5  for  k = 1: length (dancer) 6    for  j = 1:10 7     s   =  load ( sprintf ( ’%s_%d.mat’,dancer{k},j)); 8     cS  =  size (s.state,2); 9      if ( cS > 7 ) 10       i       = i + 1; 11       d{i,1}  = s.state;   %#ok<*SAGROW> 12       t{i,1}  = s.time; 13       c(i,1)  = k; 14      end 15    end 16  end 18  fprintf ( ’%d remaining data sets out of %d total.\n’,i,n) 20  for  k = 1:4 21   j =  length ( find (c==k)); 22    fprintf ( ’%7s data sets %d\n’,dancer{k},j) 23  end 25 n = i; 27  cd  .. 29   % Limit the range to 6 seconds 30 tRange = 6; 31  for  i = 1:n 32   j =  find (t{i} - t{i,1} > tRange ); 33    if ( ~ isempty (j) ) 34     d{i}(:,j(1)+1: end )= []; 35    end 36  end 38   %% Set up the network 39 numFeatures = 10;   % 4 quaternion, 3 rate gyros, 3 accelerometers 40 numHiddenUnits = 400; 41 numClasses = 4;   % Four dancers 43 layers = [ ... 44     sequenceInputLayer(numFeatures) 45     bilstmLayer(numHiddenUnits, ’OutputMode’, ’last’) 46     fullyConnectedLayer(numClasses) 47     softmaxLayer 48     classificationLayer]; 49  disp (layers) 51 options = trainingOptions( ’adam’, ... 52      ’MaxEpochs’,60, ... 53      ’GradientThreshold’,1, ... 54      ’Verbose’,0, ... 55      ’Plots’, ’training-progress’);

We then train the neural network. We use a bidirectional LSTM to classify the sequences. This is a good choice because we have access to the full sequence. For a classifier using bilstmLayer, we must set the 'outputMode' to 'last'. This is followed by a fully connected layer, a Softmax for producing normalized maximums, and finally the classification layer.

56  %% Train the network 57 nTrain  = 30; 58 kTrain = randperm(n,nTrain); 59 xTrain  = d(kTrain); 60 yTrain  = categorical(c(kTrain)); 61 net     = trainNetwork(xTrain,yTrain,layers,options); 63  %% Test the network 64 kTest   = setdiff(1:n,kTrain); 65 xTest   = d(kTest); 66 yTest   = categorical(c(kTest)); 67 yPred  = classify(net,xTest); 69  % Calculate the classification accuracy of the predictions. 70 acc         = sum(yPred == yTest)./numel(yTest); 71 disp( ’Accuracy’) 72 disp(acc);

 

 >> DancerNN  36 remaining data sets out of 40 total.     Ryoko data sets 6    Shaye data sets 10    Emily data sets 10  Matanya data sets 10   5x1 Layer array with layers:       1   ’’   Sequence Input          Sequence input with 10 dimensions       2   ’’   BiLSTM                  BiLSTM with 400 hidden units       3   ’’   Fully Connected         4 fully connected layer       4   ’’   Softmax                 softmax       5   ’’   Classification Output   crossentropyex

The training GUI is shown in Figure 7.15. It converges fairly well.

We test neural network against the unused data.

14 kTrain = randperm(n,nTrain); 15 xTrain  = d(kTrain); 16 yTrain  = categorical(c(kTrain)); 17 net     = trainNetwork(xTrain,yTrain,layers,options); 19  %% Test the network 20 kTest   = setdiff(1:n,kTrain); 21 xTest   = d(kTest); 22 yTest   = categorical(c(kTest)); 23 yPred  = classify(net,xTest);

 

 Accuracy      0.8333

Figure 7.15

Neural net training.

Table 7.3 Hardware.

The result, >  80%, is pretty good considering the limited amount of data. Four Ryoko sets were lost due to errors in data collection. It is interesting that the deep learning network could distinguish the dancers’ pirouettes. The data itself did not show any easy-to-spot differences. Calibration could have been done better to make the data more consistent between dancers. It would have been interesting to collect data on multiple days. Other experiments would be to classify pirouettes done in pointe shoes and without. We might also have had the dancers do different types of turns to see if the network could still identify the dancer.

7.11 Hardware Sources

Table 7.3 gives the hardware used in this chapter along with the prices (in US dollars) at the time of publication.