Keywords

1 Introduction

Children have to be transported safely. Securing children in a child seat is indicated. Due to structure and restraint systems, children are secured in case of an accident. Nonetheless, attention must be payed to children while driving conditions must be monitored.

A child seat can do more than just prevent injuries passively. Equipped with the right technology, injuries can be avoided, ambulance can be assisted and child’s positive mood can be preserved. Those processes are shown on the right of Fig. 1.

Fig. 1.
figure 1

NannyCaps features 

Full size image

While everybody wants to keep its child’s mood in good state, driving over long distances may lead to unnoticed conditions. On the one hand, comfort monitoring like the recognition of activity or diaper state may be a hint for a break. On the other hand, changing heart rate and respiratory rate may indicate that the child becomes scared by driving situations or multimedia systems.

Children may be left alone in the vehicle. This results in dangerous situations. The car may heat up and children suffer heat stroke [20]. If someone arrives at an accident, first aid must be provided. Subsequently, the child may be left alone in the car. Therefore, people may have to intentionally leave their child unattended [20].

All boxes on the right of Fig. 1 will likely improve the situation for children in transportation systems. By recognizing injuries and preventing injuries, supervisors can react to the child’s condition. Concerning mood preservation, a parent who intentionally left his child in the car may return early. Subsequently, an unintentionally left behind child may get attention if an occupancy recognition system alerts.

Further injuries may be caused by wrongly mounted children on child seats. For example, improper belt tightening and position changes while driving. Therefore, injury avoidance features may help. Similar to passenger out of position detection, child head position recognition may enable enhanced child seat airbags which detect out of position, too.

While injury avoidance aims to prevent injuries, injury detection shall assist medical therapy selection. Especially head acceleration may point to injuries. To assist as fast as possible, physiological conditions and extremity accelerations shall be detected by NannyCaps.

Extremity tracking may point to image processing. Nonetheless, cameras require a line of sight and distance between child and lens. Therefore, manufacturers would have to include them into their vehicles. But the system shall be a child seat which can be used in any vehicle. Furthermore, cameras can lead to privacy issues.

Physiological conditions may be measured using wristband sensors. Since people tend to forget putting on a wristband, this is fault-prone. NannyCaps is seat integrated. Hence, this problem is not present using NannyCaps.

Each of the measures on the left of Fig. 1 shall be enabled by NannyCaps. To enable an autonomous child seat, the system shall integrate all sensors into the seat. Capacitive proximity sensing (CAPS) can measure through non-conductive material. In addition, it can measure object position without touch. This enables autonomous integration.

Those measures are not directly provided by CAPS. All measures require a proper sensing electrode setup and a specific processing algorithm. As indicated in Fig. 1, not all measures are part of this paper. Focus is set on development and evaluation of head position recognition, heart rate recognition, sleep state recognition and occupancy recognition.

A concept for CAPS in child’s seat has to be designed for all those measures. Afterwards, a prototype is built following the design. Using this prototype, data of ten real world test rides (child’s seat with sensors mounted in vehicle while driving) is captured. Using this data, the designed algorithms are trained and validated. In summary, contributions are stated in the following list:

  • Determine proper sensing electrode topology to measure

    • Child seat occupancy

    • Head position

    • Heart rate

  • Design algorithms to measure occupancy, heart rate and head position

  • Prototype

    • Ordinary child seat

    • Include child seat into vehicle

    • Implementation of sensing topology and algorithms

  • Evaluation of designed systems using prototype in ten test runs

2 Related Work

2.1 Children Seat Safety Systems

As described in Sect. 1, children left behind in vehicles can lead to heat strokes [20]. Thus, a lot of research is conducted on child occupancy recognition [2, 6, 17, 23] to avoid children being left behind in the car. This shall prevent temperature induced heat strokes. Ranjan et al. [23] even use capacitive proximity sensing (CAPS) inside of a child seat.

Aneiros et al. [2] include a controller into the vehicle. The controller is capable of controlling lights, ignition switch, alarm and further vehicle components. Due to the controller, the system can inform the driver in case of a child left behind in the car. Furthermore, it can open doors and windows to decrease in-car temperature. To do so, the system requires child presence sensors. If those sensors detect an occupied child seat, an alert is started until the driver is seated. Moreover, if the delay between child seated and driver seated is greater than two minutes, the system starts the vehicle’s air condition.

Lusso et al. [17] include two force sensors and a wireless video camera into the vehicle. The force sensors detect an occupied child seat. Additionally, the video camera is directed to the child face. It shall monitor child conditions. All sensors are contained within the vehicle electrical system. Similar to Aneiros et al., Lusso et al. alert if a child is left behind. The system is tested in a real vehicle. Two temperature sensors are used to check the system behavior at different temperatures.

2.2 Capacitive Proximity Sensing Supported Children Seat Safety Concepts

Besides child conditions, a child seat position recognition system is developed by Smith [25]. Four sensing electrodes are integrated into a vehicle passenger seat. Those are used to produce 16 individual measurements. The seat may be equipped with an additional child seat. Due to the measurements, four classes of child seat position can be distinguished. Those classes are “empty”, the passenger seat is empty, “person”, there is a person on the seat, “FFCS”, there is a front facing child seat on the passenger seat and “RFCS”, there is a rear facing child seat on the passenger seat. Smith states that the system success lead to cooperation with automotive supplier NEC Automotive Electronics. NEC Automotive Electronics [16] patented a similar system.

3 Contribution

Subsections 3.1, 3.2, 3.3 and 3.4 show design and required topology for addressed measures as shown on the left in Fig. 1. Subsequently, a sensor topology is derived from those measures. The emerging topology is shown in Sect. 3.5.

3.1 Occupancy Recognition

Occupancy recognition aims to prevent heat strokes or forgotten children in child seats. Occupancy consists of two classes. The subject can be sitting on the seat which is classified as OnSeat or the seat may be empty. This is classified as NotOnSeat. This is a binary classification problem. Therefore, a decision tree classifier is used.

Since the distance between sensing electrodes and subject is assumed to take the main influence on capacitive proximity sensing (CAPS) measurement data, the feature vector of the decision tree classifier consists of the raw output of the CAPS data.

As provided by existing passenger seat occupancy systems [14], sensors below subject pelvis are required. Nonetheless, this is a minimum constraint. If further detection models require more electrodes, these are also integrated into the feature vector.

3.2 Heart Rate Recognition

As shown in Fig. 1, heart rate recognition is an essential task for emergency assistance. Moreover, changes in heart rate are induced by emotions [1]. Thus, heart rate information is helpful for mood preservation, too.

The human body consists of mostly water. Furthermore, the heart is embedded into the human body. Thus, capacitive proximity sensing (CAPS) may not enable to measure heart contraction directly. Nonetheless, measurable body displacement due to heartbeat can be discovered [18]. Body displacement affects the output value of CAPS. Therefore, NannyCaps ought to be able to measure those displacements.

Heart rate measurement is enabled if the subject is on the child seat (class OnSeat in Sect. 3.1). Therefore, heart rate recognition is only active in OnSeat state. Heart rate results from the frequency of heart contraction. Frequency becomes measurable in time series. Therefore, time series of CAPS data is deemed mandatory. Furthermore, subject back movement is expected to be low while subject is OnSeat. Thus, CAPS data of sensing electrodes in the child seat back rest is included. Even though, subject pelvis movement may not contribute to heart rate, since it is not close to subject heart, pelvis sensors may indicate vehicle vibrations. Therefore, pelvis sensors are included.

The heart rate at time t is considered as a result of a previously analyzed heart movement analysis. Thus, frequency magnitudes of ten second time intervals CAPS data is used as basis for model features. Due to the continuous nature of the heart rate, a regression model is chosen: a neural network (MLP) with one hidden layer. The hidden layer consists of ten neurons.

3.3 Head Position Recognition

As shown in [7], head injuries can be estimated from head acceleration and exposure time. Acceleration and exposure time can be derived directly from head position. Therefore, child head position recognition shall be enabled by NannyCaps.

The face center (nose) is used as target variable. Head position ought mainly to be based on sensors included in child seat head restraint. Those sensors will point to translational positions of the subject head. Different load variations in pelvis and back rest sensors are expected by different head postures. Therefore, additional sensors in child seat shall enable head rotation estimation.

Head restraint capacitive proximity sensing (CAPS) value difference is expected to be the main indicator for head displacement. Thus, the feature vector for the regression model is based on the difference between each CAPS output. For example, this would mean that a feature vector entry is formed by the difference between left and right head restraint sensor output. Due to the continuous nature of head position, a random forest regression model is selected. Head position recognition shall be enabled by training a model. Subsequently, the model is tested with unseen data.

3.4 Sleep State Recognition

Actigraphy is used to distinguish between sleep phases [19]. It is a measure for mapping subject movement to sleep phases. Furthermore, respiratory rate monitoring is included in sleep analysis. Capacitive proximity sensing (CAPS) is applied to track respiratory rate in office furniture in [4]. Furthermore, it is used to track subject activity in [4, 5, 13] or extremities in [9,10,11,12]. Since those two measures point to sleep phases, CAPS is used in child seat to distinguish between asleep and awake child in child seat.

Similar to heart rate recognition in Sect. 3.2, frequencies of CAPS data are used. Therefore, the frequency magnitudes of a time window of ten seconds CAPS data is used. Thus, comparability between heart rate recognition and SR performance shall be enabled. Subsequently, the same CAPS topology as in Sect. 3.2 is deemed appropriate. Nonetheless, to increase activity recognition, leg restraint included CAPS sensing electrodes shall be mounted.

3.5 Sensor Topology

Sections 3.1, 3.2, 3.3 and 3.4 name several mandatory sensing electrode positions. Measurement at subject pelvis is required for occupancy recognition, measurement at back rest is required for heart rate recognition, measurement at head restraint is required for head position recognition and additional sensors at leg restraint are required for sleep state recognition.

The spatial electrode topology is derived of those requirements as shown in Fig. 2. Sensing electrode positions are marked by blue circles. All sensors shall be included under child seat cushion. If the provided cushion is not sufficient for placing electrodes, the electrodes shall be mounted under non-conductive parts of the seat structure.

Fig. 2.
figure 2

Designed sensor topology 

Full size image

4 Implementation

In advance of putting designed sensing topology as shown in Sect. 3.5 into practice in Sect. 4.2, an appropriate sensor system is selected in Sect. 4.1. Furthermore, additional sensors are added to generate labeled data. The measurement setup to monitor test rides and generate labels for capacitive proximity sensing data is presented in Subsect. 4.3.

4.1 Capacitive Proximity Sensor Selection

Following design in Sect. 3.5, eight capacitive proximity sensing (CAPS) electrodes are required to map NannyCaps concept to a prototype. Since real systems provide discrete data, an appropriate sampling rate has to be determined. Obviously, heart rate is a frequency based biometric characteristic. Hence, CAPS sampling rate selection is based on heart rate recognition.

Test ride subject age is about 1.5 years. Therefore, the heart rate range is in an interval between 98 to 140 beats per minute (bpm) while awake and 80 to 120 bpm while asleep [8]. Mapped to beats per second (bps), heart rate should be between 1.33 and 2.33 bps (\(HR_{max}\)). This has to be covered by the heart rate recognition application.

The selected CAPS device is capable to monitor eight sensing electrodes [15]. Each sensing channel output is sampled 25 Hz. This results in an oversampling factor of 10.7 compared to \(HR_{max}\). Therefore, the Nyquist Rate is exceeded [22]. Unprocessed circuit boards (copper, epoxy) are used as CAPS electrodes and shielding. Each electrode is of 10 cm length and 16 cm width. This results in an area of \(0.016\,\mathrm{m}^2\). An equal electrode layout is selected to minimize deviating temperature effects. Due to the setup of circuit boards, CAPS electrodes and shielding area are congruent in sensing direction.

4.2 Child Seat Integration

The provided eight channels of the capacitive proximity sensing (CAPS) device and the attached circuit boards are mounted on an ordinary child seat, shown in Fig. 3. As shown in Fig. 3, seat cushion is removed completely.

Fig. 3.
figure 3

Used child seat with sensors (Color figure online)

Full size image

Afterwards, the sensing electrodes are placed according to Sect. 3.5 (Fig. 3, yellow stars). The available space at the leg restraint was not sufficient for electrode mounting. As shown in Fig. 3 (right), leg related electrodes are placed under seat skeleton. Then the seat cushion is reinstalled on the seat (Fig. 4).

4.3 Test Ride Surveillance System

The test ride surveillance system must capture the activity, occupancy and heart rate of the subject. Concerning heart rate measurement, an optical heart rate sensor [24] is used. The sensor is labeled in Fig. 4 with OH1. Occupancy recognition, head position recognition and sleep state recognition are labeled by using image processing. Images are provided by a camera. The camera is attached to the vehicle interior roof. It points towards the child seat. Figure 4 shows the camera view.

Fig. 4.
figure 4

Test setup camera view cut out 

Full size image

5 Evaluation

5.1 Captured Data

Captured data is comprised of ten test rides including a heart rate sensor, capacitive proximity sensing and a camera. More than nine hours and more than 600 km are recorded. Video data is analyzed manually to label occupancy and sleep state. Figure 5 shows the distribution of those states among test rides. Awake and asleep state show approximately balanced duration. An imbalanced duration for on seat and not on seat is indicated by a ratio of approximately 36 to one.

Fig. 5.
figure 5

Subject states during measurements diagram 

Full size image

5.2 Occupancy Recognition

Transition from subject sitting on seat (OnSeat) and not on seat (NotOnSeat) is fuzzy. Therefore, a buffer of ten seconds before and after each transition is excluded. 8.6 h OnSeat and 16.5 min NotOnSeat data is collected. As described in Sect. 3.1, all eight channels raw data is used as model input vector. The complete dataset consists of 776000 OnSeat and 25000 NotOnSeat samples.

Those samples are split randomly into 50% training and 50% test data. Afterwards, a decision tree classifier is trained using training data to distinguish between OnSeat and NotOnSeat. Subsequently, labels for test data are predicted by the classifier. The evaluation of test data results in a true (T) positive (P) rate of \(\approx \)1 (P: 387057, TP: 387047) and a true negative (N) rate of \(\approx \)1 (N: 12457, TN: 12444).

5.3 Heart Rate Recognition

Heart rate recognition shall be facilitated using data of child seat integrated capacitive proximity sensing (CAPS). Thus, the data of each test ride is filtered for the subject being on the seat. Subsequently, CAPS data with label OnSeat, as shown in Sect. 5.2 is concerned. Except test ride one, all test rides have one single OnSeat phase. As shown in Fig. 5, test ride one hast two OnSeat phases. Since the first phase covers less than four minutes, this phase is omitted. Captured heart rate data are synchronized with CAPS data via time stamp information.

CAPS data is processed according to Sect. 3.2. Thus, a frequency spectrum remains. Figure 6 visualizes the complete dataset. It is grouped by heart rate data. Afterwards, frequency magnitudes of each group are averaged. This is repeated for all channels. Each circular sector represents one channel beginning from the channel label to the next label in counterclockwise direction. The angle of each sector represents the frequency. For example, frequencies of channel one range from zero Hz at label “Ch1” (angle = 45\(^\circ \)) to 12.5 Hz at label “Ch2” (angle = 90\(^\circ \)). Colors represent the MinMax normalized heart rate value.

Fig. 6.
figure 6

Heart rate recognition feature vector data 

Full size image

According to Sect. 3.2, a neural network regression model is trained and evaluated. Data is split into 75% training and 25% test data. Subsequently, training data is resampled to show equal value distribution. A mean absolute error of 6.2 bpm is measured. A high difference between training performance (R\(^2\) \(\approx \) 0.85) and test performance (R\(^2\) \(\approx \) 0.55) points to overfitting.

Fig. 7.
figure 7

Head position labeling 

Full size image

5.4 Head Position Recognition

Image processing, in particular, face recognition is used to retrieve horizontal head position [21]. A sample recognized face area is shown in Fig. 7. Since relative position of camera and child seat is not fixed, a child seat fixed reference point is added. A label on the child seat is used as stable reference point. Reference points are collected from random image samples. Therefore, influence by child seat movement due to test ride progress (subject movements, vehicle vibrations, rotation due to winding road) ought to be minimized compared to one single reference point.

The used face recognition algorithm is not robust during the whole measurement. Therefore, recognized face position is filtered. Only recognized face areas, within region of interest (Fig. 7: “ROI”), are used. The remaining detected faces are then filtered according to their area. Only areas between 35th and 65th percentile of all areas remain. Recognized horizontal face position is subtracted from reference point position. Afterwards, half of the face areas width is added to retrieve face center point. This results in the target label as shown in Fig. 7.

The used face recognition algorithm does not work during all lighting conditions. Furthermore, only frontal faces are detected and face profile is not tracked. To include more labels, especially from positions at the seating limits, manual labels are added. Those manual labels consist of the subject’s nose position.

The labeled data is processed as described in Sect. 3.3. Using this dataset, a random forest regression model is trained and tested. Training and testing are conducted by use of tenfold cross validation. A mean coefficient of determination (R\(^2\)) of 0.95 is measured. Furthermore, the mean absolute error (MAE) is 3.87 pixels.

5.5 Sleep State Recognition

Data is labeled as presented in Sect. 5.1. 261.61 min asleep and 185.62 min awake are recorded. Due to the fuzzy transition between asleep and awake, a buffer of 20 s at each transition is extracted. As presented in Sect. 3.4, comparable features to heart rate recognition are generated. In particular, a time window of 10 s (250 samples) is used. Thus, the resulting model dataset is comprised of 1235 awake (P) and 1737 asleep (N) samples.

Using this dataset, a random forest classifier is trained. Data of all test runs is split into 80% training and 20% test data. Prior to training, training data is balanced. A true (T) positive (P) rate of \(\approx \)0.93 (P: 242, TP: 226) and a true negative (N) rate of \(\approx \)0.91 (N: 319, TN: 291) is measured evaluating test data.

6 Discussion

NannyCaps capabilities enabled due to sensor topology and integration are covered in Sect. 6.1. A view on data preprocessing and statistical model fitting and possible dependencies between model targets is presented in Sect. 6.2. In Sect. 6.3, the significance of exercised measurements and the prototype setup is discussed.

6.1 Sensor Topology and Invisible Integration

Due to the capabilities of capacitive proximity sensing (CAPS), invisible integration into the child seat pan is enabled. CAPS electrodes are mounted under the seat cushion. This is enabled by non-conductive materials. Hence, the system can be embedded autonomously into child seats. Additionally, the subject is always closest to sensing electrodes. Thus, the system is not affected by objects between subject and sensing electrodes. Hence, robustness of sensing is increased compared to systems which require line of sight.

While there may be no object between sensing electrodes and subject (except child seat cushion), changing temperature and moisture conditions take influence on system robustness. Moisture may shadow the subject or at least change the sensors offset. Recognition models, which rely on static sensor data, could be affected. NannyCaps occupancy recognition relies on static data. Wet cushion or changing temperature could lead to false predictions.

6.2 Model Performance and Data Interdependence

Fig. 8.
figure 8

Heart rate and sleep state distribution 

Full size image

A violin plot is shown in Fig. 8. A violin plot has similarities with a box plot. The occurrence of data points for the respective label is shown as outlined areas in blue and red. Median values are represented by a white point in the middle of each violin. Furthermore, inter-quartile range (IQR) is plotted as a black thick vertical bar. Values within first quartile have a value less than IQR. Values which are not within third quartile are greater than IQR. A range between first quartile minus 1.5 times IQR and third quartile plus 1.5 times IQR is presented by the thin vertical lines. Outliers may be indicated by values greater or less than this line.

As shown in Fig. 8, the median heart rate values are: Awake about 122 beats per minute (bpm), Asleep about 105 bpm. Furthermore, no heart rate intersection is shown within IQR. Nonetheless, data intersections between asleep and awake are indicated. Those intersections are not considered to be outliers.

Due to the data distribution in Fig. 8, heart rate recognition may be based on similar model observations like sleep state recognition. Although sleep state recognition results imply further observations which distinguish between awake and asleep at the same heart rate.

A correlation between sleep state recognition and heart rate seems plausible for healthy humans. Therefore, the correlation in Fig. 8 seems to be valid. Thus, the assumption that heart rate recognition is mainly based on sleep state recognition cannot be discarded. Nonetheless, heart rate recognition could be improved by additional sleep state recognition data. Still, sleep state recognition could be improved by additional heart rate recognition data. In both cases, models could benefit from the respective observations. Contrary to heart rate recognition, head position recognition, occupancy recognition and sleep state recognition show good performance.

6.3 Implementation and Measurements

A prototype is successfully built based on design in Sect. 3.5. Furthermore, environmental measurement systems were installed in an ordinary vehicle. The conduct of ten test runs shows plenty of variations in environmental conditions like lighting and interior temperature. Due to conduct in summer, the measurements show large interior temperature ranges. Nonetheless, system evaluation would benefit of winter measurements or moisture simulation as well.

Labels for head position recognition are derived from captured images. Those images are partially captured automatically. Thus, the labels themselves show measurement errors. Even though acceptable performance is shown in head position recognition evaluation, a more robust labeling system would improve performance. Those labeling systems could be comprised of depth cameras or special hat markers.

The test series was conducted with one minor subject. While parents provided a letter of agreement here, future tests may include diaper checking. Thus this could lead to an unusual stress for minors while testing. Hence, an ethics committee would have to check this experiment.

7 Conclusion and Future Work

Considering selected measures as shown on the left of Fig. 1, concepts for head position recognition, heart rate recognition, sleep state recognition and occupancy recognition are defined (Sect. 3). The built prototype (Sect. 4) is based on those concepts. An ordinary child seat is used. Within this child seat, all capacitive proximity sensing (CAPS) electrodes are included enabling invisible integration into an autonomous system. Nonetheless, system autonomy is just an idea, yet. Thus, in the future, concepts for an autonomous system have to be defined. A cellular module could be included for communication.

The built prototype and additional sensors are used to collect data in ten real world test rides. Using this data, the defined concepts are trained and evaluated (Sect. 5). Even though environmental conditions vary within those test runs, more data of different subjects, different vehicles and different child seats has to be collected. This shall refine evaluation findings.

Heart Rate Recognition. The designed heart rate recognition model shows a mean absolute error of approximately \(\approx \)6.2 bpm. Even though an error of \(\approx \)6.2 bpm seems to be acceptable for an almost not tuned model, an inconclusive verdict about heart rate recognition is shown in evaluation and argued in discussion. This is enhanced by a coefficient of determination \(\approx \)0.5.

Two processes could evaluate heart rate recognition with certainty. On the one hand, the measurement has to be extended to different subjects. More subjects may have different heart rates at rest and during activity. Different heart rate characteristics in similar movement may eliminate direct sleep state to heart rate correlation. This may also improve model performance. On the other hand, the sensing topology could be extended to include sensing electrodes into the seat belt. This configuration would enable shunt or transmit mode sensing [25]. Sensing electrodes close to subjects’ breast and back could enable to filter body movement from micro-movements as shown by Michahelles et al. [18].

Respiratory Rate Recognition. Those sensing electrodes within seat belt could enable respiratory rate recognition. Similar to heart rate recognition, respiratory rate recognition is assumed to be important for a child seat monitoring system. Thus, it is included in intended NannyCaps measures (as shown in Fig. 1). CAPS systems are already used for respiratory rate checking in office furniture [4]. Additionally, respiratory emissions like yawning are detected in automotive seats [3]. Therefore, this feature seems to be practicable in a child seat. To get the ground truth, a respiratory rate monitoring system, like a face mask, would have to be included in future measurements.

Diaper State Checking. One useful feature for a smart child seat would be diaper state checking. While this may lead to delicate topics, information about diaper state could help parents to protect their children skin and interpret baby’s articulations. To tackle this feature, measurements with diaper moisture sensors would have to be included in future measurements.