Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Since walking is one of the most frequent and important human activities, much attention has been paid to the analysis of gait patterns [1] in the context of developing smart Ambient Assisted Living solutions [2]. The investigation of gait parameters as well as motion sequences reveals valuable information about the individual mobility, the daily expenditure level and health stability of a person. Furthermore several specific diseases like diabetes, depression and peripheral neuropathy are correlated with symptoms characterized by gait instability or unsteadiness, which are indicated by a shorter step length [2]. However, much of the work concerning the observation of gait properties focuses on wearable devices. Marschollek et al. [3], try to assess the risk of a fall for an elderly person using an acceleration sensor attached to the waist by a custom belt. In contrast, our work focuses on pure passively operating sensor systems, and therefore provides a non-invasive permanent monitoring process of a person’s well-being. The principle idea is illustrated in Fig. 1.

Fig. 1
figure 1

Principle of vibroacoustic monitoring

Full size image

2 Principles of Vibroacoustic Monitoring

We are interested in extracting two relevant pieces of information from the sensor signals as described below. First, we would like to determine the cause of the vibration signal which allows us to discriminate between signals of interest like a footstep of a person and undesired noise sources like washing machines or walking frames. This requires the implementation of a classification algorithm. Second, the exact location of a seismic event is of great interest. Under the constraint of a purely passive operation mode there are two basic methods of determining the position of a seismic event.

The first method (Fig. 2) is an extension of the conventional TDOA (Time Difference of Arrival) approach, which is well known in the processing of airborne signals. However, this method suffers from diverging waveforms caused by dispersion and therefore only produces robust results in non-dispersive media like air. By taking the dispersion characteristics of the underlying medium into account, we are estimating the so called range differences between every pair of sensors in a direct way and subsequently formulate the nonlinear system of equations that describes the localization problem in a mathematical way [4]. It is worth noting that this approach makes use of uniaxial sensors i.e. it uses the vertical acceleration of floor-surface-particles (z-axis).

Fig. 2
figure 2

Illustration of TDOA-approach

Full size image

In contrast, the second approach (Fig. 3), which is called AOA (Angle of Arrival)-approach, uses tri-axial sensors and consequently processes the acquired surface wave (Rayleigh Wave) as a vector-wave represented by the acceleration of the particles into x-, y- and z-direction of the Cartesian sensor coordinate system. The interpretation of these three components reveals the angle at which a surface wave impinges at the sensor [5]. We compute the exact location of the seismic event by triangulation using two or more tri-axial sensors.

Fig. 3
figure 3

Illustration of AOA-approach

Full size image

After this short review of existing localization procedures, we present a novel idea on the development of a localization-scheme making use of only one tri-axial sensor station (Fig. 4). The key idea behind this scheme, besides the usual AOA-estimation process, is to exploit the effect of dispersion in order to estimate the distance to the seismic source. To accomplish this, we use advanced time–frequency transformations [6], which allow a frequency-dependent time of arrival estimation. Knowing the dispersion curve it is possible to translate this information into an estimated distance and in combination with the determined AOA into the position of the seismic source.

Fig. 4
figure 4

Illustration of single-station approach

Full size image

Localization in a continuous manner becomes a problem of tracking which offers e.g. the extraction of motion patterns. Another processing of the location-information allows the distinction between more than one entities of the same class (e.g. the distinction between footsteps stemming from different persons) by using probabilistic models representing the transition probabilities as a function of space.

3 Challenges

3.1 Multipath Propagation

A robust operation of the developed algorithms assumes the signal to be noiseless and stemming from a single-mode Rayleigh wave. In a real indoor environment however, we expect disturbances in the form of body wave reflections stemming from the bottom side of the floor as well as Rayleigh wave reflections from bordering walls (Fig. 5).

Fig. 5
figure 5

Illustration of wave propagation

Full size image

3.2 Dispersion

In vertically layered media, Rayleigh wave propagation is characterized by dispersion, meaning the propagation speed is frequency dependent. This leads to diverging waveforms (Fig. 6) and makes conventional localization methods from air borne sound signal processing inapplicable because they generally do not include the effect of dispersion in their signal model. Therefore we intend to reliably estimate the dispersion curve of the floor in order to improve localization results.

Fig. 6
figure 6

Illustration of dispersion

Full size image

3.3 Space Variance

In general we have to consider a floor as an inhomogeneous anisotropic propagation medium because parts of the floor construction, room furniture or other objects related to floor properties do affect Rayleigh-wave propagation. This implicates dispersion behaviour which is dependent on the excitation point and differs for propagation into different directions. Consequently, consideration of the dispersion effect into algorithm will require a manageable model of the floors space variant properties.

3.4 Noise

We not only expect noise sources like road noise but also interactions with objects such as doors or kitchen furniture cause disturbing vibrations that need to be ignored in the localization process. Moreover, sound pressure originating from TV, radio or other participants cause vibrations at the sensor that influence signal evaluation [2].

4 Results

Figure 7 shows the measurement setup for evaluating the localization performance of the TDOA-approach. A typical parquet floor of size 6.9 × 5.57 m was excited by an impulse hammer at ten different positions, five times each. Floor vibrations where gathered by four IMI Sensors with a sensitivity of 100 mV/g and a resolution of 350 μg. Table 1 contains the results we gathered by applying a conventional adaptive threshold based TDOA-algorithm. We assumed the velocity of the fastest occurring frequency to be 1,150 m/s. Localization results vary considerably for different excitation points. Moreover these results show the performance without taking dispersion into account.

Fig. 7
figure 7

Measurement setup for localizing impact hammers

Full size image
Table 1 Results of impact hammer localization TDOA-approach
Full size table

We tried to deploy the TDOA based localization algorithm in order to localize real footstep signals. Figure 8 shows a screenshot of a demo video, which should demonstrate the algorithms operational functionality. In the left part of the picture you can see the original sequence—a female person walking over the floor. The right part illustrates the estimated footstep-locations as a walking path. As can be seen in Fig. 8 we were able to successfully follow the path of the person. However, reasonable localization results are limited to steady walking people with appropriate footwear.

Fig. 8
figure 8

Screenshot of demo-video

Full size image

Further results refer to the determination of the AOA approach a wave impinging at a tri-axial sensor station. Figure 9 shows the measurement setup, where a PCB 356A17 sensor was placed on a concrete floor, not too close to any floor borders. Every test-angle was excited five times with an impulse hammer to compute an average mean error of the estimated angles, which are presented in the left part of Table 2a. Next, we placed the sensor directly a few centimeters to a wall and repeated the experiment. The results can be seen in the right part of Table 2b. Apparently, this sensor position leads to a much higher mean error of the estimated angles. We conclude that there are reflections from the wall, sumperimposing with the direct signal and degrading bearing-estimation results.

Fig. 9
figure 9

Measurement setup for determining angle of impinging wave

Full size image
Table 2 Results of impact hammer localization AOA-approach
Full size table

These promising results led us to the decision to further follow the approach using tri-axial sensor technology and AOA-estimation by testing it on real footstep data. Figure 10 illustrates the experiment’s setup, where again a PCB 356A17 sensor station was stimulated by 15 barefooted heel strikes of medium to low energy at each angle. Corresponding AOA-estimation results can be seen in Table 3 and show reasonable accuracy.

Fig. 10
figure 10

Measurement setup for the evaluation of AOA-algorithm on real footstep (heel strike) signals

Full size image
Table 3 Results of heel strike localization AOA-approach
Full size table

4.1 Real-Time Footstep Detection and Localization System

In order to further test the developed algorithms in real life scenarios, we are implementing a real-time footstep detection and localization system. Its overall structure is depicted in Fig. 11.

Fig. 11
figure 11

Overall structure of real time vibroacoustic monitoring system

Full size image

First we have to make sure to detect what we are interested in, namely footsteps. Figure 12 shows sample footstep signals stemming from a person walking barefooted nearby the sensor on a laminate floor in a steady motion. The steps are characterized by broadband high frequency contents stemming from toes and tangential interaction with the floor-surface as well as low frequency content produced by the heel-strike. Due to the strong damping of high frequency components, we focused on detecting the low frequency content, propagating over longer distances. Therefore we decided to implement a heel-strike classifier in order to detect footsteps. To this end we compute Mel Frequency Cepstral Coefficients (MFCCs) in addition to a general low frequency content energy feature. Even though MFCCs are known to have been developed for speech processing algorithms and may not be the best choice for this classification task, they are able to characterize the relatively short heel-strike signal rather compactly. After further reducing the dimensionality of feature space by Principal Component Analysis (PCA), a Support Vector Machine (SVM) classifier decides whether the observed signal block contains the heel-strike of a footstep signal.

Fig. 12
figure 12

Exemplary footstep signals

Full size image

The detected footstep signals are processed by the localization stage, which consists of AOA-estimation for every tri-axial sensor station and finally the determination of source location by triangulation. To further increase robustness, the whole estimation is carried out three times with slightly shifted windows and filtered through a median function to accommodate for classification inaccuracies. During the development of the AOA-estimation algorithm based on Zhang [5], we tried different implementations in order to achieve the most robust and reliable result on real footstep signals. The most promising one is described as follows.

First of all the extracted heel strike windows of the three axes are transformed into frequency domain by a conventional Fast Fourier Transform.

$$ A_{z} = fft(a_{z} ) $$
(1)
$$ A_{y} = fft(a_{y} ) $$
$$ A_{x} = fft(a_{x} ) $$

Because of the elliptical motion of floor-surface particles we perform a 90 degree phase shift on the z-axis of the acceleration signal which can be obtained by using the imaginary part of a Hilbert Transform

$$ A_{zs} = A_{z} e^{{ - j\frac{\pi }{2}}} $$
(2)

In order to compute the frequency dependent cross correlation of the z-axis and the y-axis respectively x-axis to

$$ r_{zx} = A_{x} * conj(A_{zs} ) $$
(3)
$$ r_{zy} = A_{y} * conj\left( {A_{zs} } \right) $$

By applying the arcus tangent function to these signals, which represent the on the y-axis and x-axis projected compressional part of the surface wave we obtain the frequency dependent angle function.

$$ \phi = \arctan (\frac{{r_{zx} }}{{r_{zy} }}) $$
(4)

Zhang [5] developed a weighted averaging method in order to compute the estimated angle out of the obtained function in a way that weights frequency components according to their energy. However, this approach did not work very well in our conditions so we developed an alternative approach for the final angle computation.

Due to the upper limit on reasonable phase accuracy of the used sensors we restricted the analysed frequency content to the range between 20 and 2,000 Hz. Within this band, the frequency dependent angle function is then windowed using overlapping rectangular windows. For each window the variance is calculated and the ones with the three lowest variance values are selected. Only frequency components exceeding an adaptive energy threshold are considered. For each selected window the median is calculated and eventually the median out of these three values leads to our final angle estimation. Should either the energy level of the signal be too low or the variance of estimations too high, a reject option can be implemented to avoid non-robust results.

First experiments with the system showed that reasonable results strongly rely on a sufficient SNR of the footstep signals.

5 Applications

We assess that the introduced technology has great potential to be deployed in Ambient Assisted Living and Surveillance solutions.

  • Activity-Recognition/Quantification: How mobile is a monitored person? Ratio Activity/Passivity?

  • Abnormality-Recognition: Falls, falling objects

  • Determination of gait parameters as indications for pathological gait patterns and/or fall risk assessment

  • Localization/Tracking: Analysis of motion patterns, Multi-Person-Recognition

In addition to these functions, the idea of monitored persons consciously interacting with the system reveals other remarkable application scenarios in which the system is used as a tangible acoustics interface (TAI) [7]. In this way, elderly people could interact with any solid object to communicate their needs for example by furnishing the table of the living room with our system. By discriminating different types of excitation activities even more interactive applications with input specific responses are imaginable. First experiments exploring this point of view were limited to the localization of finger tips on table surfaces. They showed general operational functionality of localization but a severe degradation of achievable accuracy caused by reflections of table boundaries. First ideas to overcome this problem include the application of windowing techniques as well as the construction of appropriate table shapes.

6 Future Work

Future developments will concentrate on the evaluation of the whole classification and localization system on real footstep data. Furthermore, because of the transient character of the signals we would like to use more appropriate features like the coefficients of the discrete wavelet transform for the heel strike classification task. We also want to further explore the single station localization approach. Therefore we will thoroughly study a STFT-based approach offered by Chun [4] in order to estimate the dispersion curve of floors and to introduce the obtained dispersion curve into our localization algorithms. Finally, we will carry out further experiments on tables in order to investigate the disturbing reflections from table boundaries currently leading to worse localization accuracy.

7 Conclusion

We introduced the concept of vibroacoustic monitoring by means of classification and localization of seismic events. Focusing on the localization part, after reviewing existing localization methods, we presented a novel idea using only one sensor station. The proposed method minimizes installation cost without significantly increasing the financial cost. However, more work has to be done in order to be able to apply the theoretical idea in a real application scenario. Besides that, AOA-estimation of two sensor stations followed by triangulation seems to be the most promising approach. A robust localization result depends on how well we are able to account for multipath propagation as well as the inhomogeneity and anisotropy of floors as an underlying medium. Furthermore the deployment of the described technology in a real world application will raise the need to account for external noise sources in order to ensure operational localization functionality. The purely passive functionality as well as the low installation cost makes the introduced non-invasive technology easily deployable in several potential application scenarios and therefore interesting for further research.