Embedded Smarthouse Bathroom Entertainment Systems for Improving Quality of Life

Abstract

The phrase “Smarthouse to improve the smartness of a human’s daily life” has two meanings. One is to improve individual smartness, which represents the Quality of Life (QoL); the other is to improve social smartness, which includes human communications and social consumptions. This chapter primarily describes entertainment systems that can be embedded particularly in the bathrooms of smarthouses and used by humans in everyday life to improve QoL. The systems include “Bathonify,” a sonification system that reflects the bathing states and vital signs of the bather; “TubTouch,” a bathtub entertainment system that uses embedded touch sensors and a projector to control various equipment and systems; and “Bathcratch,” a DJ scratching music system that is operated by rubbing and touching the bathtub. Even though these systems are based on Japanese bathing culture and style, they provide advances in the pleasures of everyday life. In addition, these embedded systems and their techniques provide advances in computer entertainment platforms that can be extended to various places and situations.

6.1 Introduction

Food, clothing, and shelter (housing) are essential for the survival of human beings, and thus, research has been actively underway in these areas. With network-enabled household appliances now a reality, ubiquitous computing (Weiser 1991) research to enhance the convenience and comfort of everyday life in the home, resulting in the term “Smarthouse,” is also actively being conducted (Mason et al. 1983; Kidd et al. 1999; Intille et al. 2005; Ruyter et al. 2005; Ueda and Yamazaki 2007; Siio et al. 2010; Hirai and Ueda 2011). Some of the research being conducted is focused on entertainment aspects and includes various ideas and systems for enjoying everyday life. To date, most of the research has focused on the living room and kitchen, where electrical appliances are easily installed. In contrast, it is difficult to equip the bathroom to become a computing environment because the sharp changes in temperature and humidity that occur there make installation of equipment difficult. Despite this difficulty, however, a Japanese bathroom may have a variety of electronic equipment such as a heater/dryer unit in the ceiling, an external water heater, equipment for remotely controlling bathroom temperature, water temperature, and water quantity, and a speakerphone between the bathroom and the kitchen. Further, we are beginning to see other value-added amenities and equipment such as TV and audio systems, lights with dimmers, Jacuzzis, and mist generators in bathrooms. Thus, the bathroom can now be viewed as another space to bring various ubiquitous computing technologies and to enhance the everyday act of bathing so that it is more entertaining even though the equipment may depend on the Japanese bathing culture and the style of the home.

This chapter discusses a number of systems in terms of their entertainment and practical applications for the Japanese-style bathroom that are controlled via various embedded sensors. The first system discussed is Bathonify (Hirai et al. 2004), an interactive sonification system that reflects a bather’s actions and vital signs while in the bathtub using an external water heater and embedded ECG sensors in the bathtub. It uses sonified sounds and music to create an amenity space in the bathroom that bathers can enjoy and other family members in the home can utilize to listen and monitor the bather’s state, actions, and vital signs in real time. The next system discussed is TubTouch (Sakakibara et al. 2013; Hirai et al. 2013), which uses embedded capacitive touch sensors to convert the edge of the bathtub into a user interface. TubTouch can operate various types of bathroom equipment and also enables all age groups, from children to elderly people, to have access to a variety of entertaining applications while bathing. The final system discussed is Bathcratch (Hirai et al. 2012), which utilizes an embedded piezo sensor to enable bathers to play a DJ scratching music entertainment system by rubbing the edge of the bathtub. Works related to these three systems, their practicability as regards bathroom facilities, common grounds of all the systems, interactivity of each system, and improvements brought to Quality of Life (QoL) are subsequently discussed.

6.2 Bathonify: Sonification System to Reflect Bather’s Motion and Vital Signs

6.2.1 Concept Underlying Bathonify

The goal of Bathonify (Hirai et al. 2004) is to create an amenity space that is both unobtrusive and enjoyable, and which can thus facilitate novel bath systems based on ubiquitous computing. The idea is to use bathwater as a natural ambient medium and express the state of the water through sound. The Bathonify system sonifies the changes in water level and ripples on the surface of the water as interactive sounds (sound effects and music) utilizing a water pressure sensor equipped in an external water heater. In this system, bathwater coupled with sound enables a new level of active interaction.

To further expand the expression of sound during quiet bathing (minimal movement), this system takes account of the bather’s vital signs. For everyday computing, it is important for this system to show health management information such as the bather’s physiological and psychological states and to keep the bather informed of those states. To collect vital signs, the system utilizes a hidden interface that unobtrusively measures these states when the bather is submerged, without attaching any sensors to the body. In addition, the system processes the sound made by vital signs such as breathing changes and heartbeats even in a quiet bathing state (no movement). One condition to keep in mind is that heartbeats tend to rise as the bathing period lengthens because of water pressure and heat stress on the body. The Bathonify system uses this tendency to correlate the music tempo with the beating of the heart. The bather can thus be aware of his/her health and changing body state through the tempo of the music.

This novel bathing environment enables a bather to experience auditory pleasure as well as monitor vital signs in an unobtrusive manner. However, it must be kept in mind that sound design is one of the most important factors in the seamless coupling of functionality and ambient media.

6.2.2 System Overview and Design

Figure 6.1 gives an overview of the Bathonify system along with the embedded features described in the previous section. The information and technology required for each feature are described below.

Fig. 6.1

Overview of the Bathonify system

6.2.2.1 Measuring Movement Information

In the Bathonify system, a water pressure sensor embedded in a fully-automatic bathwater heater measures the movement of water inside the bathtub. This sensor only checks the water level periodically; thus, it is idle most of the time. By continuously measuring the water pressure, we can measure changes in the water level as well as monitor water surface ripples and dynamic pressure changes in the bathwater. This type of sensor is already present in bathing equipment and is very safe against electric shock in watery environments. The sensor itself resides inside the water heater, which provides a hidden interface out of the bather’s view. We consider that a bather can influence water movement in the following ways:

• 1. Stir the water in the bathtub to eliminate temperature differences.

• 2. Ladle water from a wash bowl or a bucket (small movement).

• 3. Become submerged in the bathwater (resulting in the water level rising).

• 4. Move arms around in the bathtub.

• 5. Exit the bathtub (resulting in the water level lowering).

Figure 6.2 shows the output data from a water pressure sensor (model KH0578-30 by Hamamatsu Photonics) for an actual bather based on the movements described above. The measurement data for a water heater contains circuit noise that has a higher frequency than both bather movements and water movements in the bathtub. As a result, the noise can easily be eliminated with a low-pass filter. A comparison of Figs. 6.2a and b shows that stirring the water strongly causes a large change in the pressure of the water. Figure 6.2c shows how the water pressure slowly rises as the bather enters the bath, starting from the feet and submerging the body up to the shoulders. Conversely, Fig. 6.2f shows how the water pressure is suddenly lowered as the bather rapidly exits the bath. Figure 6.2e shows that the pressure changes only a little compared to when the water is strongly stirred without a bather in the bathtub. This difference may be because the sensor cannot easily detect large stirring motions on the surface, and thus changes in water pressure, when a bather is in close proximity to the opening of the hot water supply. These measurement results draw attention to the importance of considering differences in ripple amplitude when a bathtub is occupied and unoccupied.

Fig. 6.2

Water pressure sensor output for each situation

6.2.2.2 Measuring Vital Signs

We use the water pressure sensor to take breathing measurements. The act of breathing causes the water level to fluctuate by a few millimeters in a bathtub. Therefore, the sensor extracts information about each breath from variations in the water level. Preparatory tests showed that low-pass filter processing with a cutoff at 0.35 to 0.4 Hz gives virtually the same results as a specialized breathing measurement instrument (model AE-280 S by Minato Medical Science). Figure 6.3 compares the water volume to the respiration volume. The upper of Fig. 6.3 shows a graph of conversion values for the bathtub volume (including breathing volume) after low-pass filter processing of the water level values with the cutoff set at 0.35 Hz. From these values we can extract a breathing curve using low-pass filter processing and couple the breathing depth information with sound control. However, this processing only works when a person is bathing. If the water surface undulates a lot, the sensor will extract other information in addition to the breathing component. Consequently, breathing processing is limited to relatively mild undulations on the surface of the water.

Fig. 6.3

Comparison of respiration curves (Upper) by water pressure, (Lower) flowmeter

This system also measures heartbeats in real time while bathing. The bathtub is a conventional bathtub with small electrodes, which provide a contactless method to measure heartbeats when a bather enters the bathtub, attached to its sidewalls. The heartbeat measurement circuit eliminates noise and samples an amplified waveform at 8 bit/kHz using an RS-232 C connection. Figure 6.4 shows an actual electrocardiogram (ECG) waveform from a bather bathing in a bathtub.

Fig. 6.4

ECG measured in a bathtub

6.2.2.3 Extracting and Processing Control Information

From the movement information and vital signs, we extract the parameters and events needed to perform sound control processing. To control the type of sound, volume, and production conditions, the following states and parameters are used: entering the bath, exiting the bath, amplitude of water surface ripple, instantaneous heartbeat based on R waves, R wave height, and T wave height. To control sound production timing, events to detect triggers for R waves and T waves are used, and trigger signals for water surface ripples above a threshold value are also used. Pretreatment functions for measurement data are primarily handled by the microprocessor of the water heater unit. However, if the water heater is connected to the Internet through a home LAN, the water heater can send the measurement data to external processors for pre- and post-processing.

6.2.2.4 Sound Control

The sound controller uses parameters and events obtained from processing to extract control information. Settings are coupled with parameter and event information for output sound such as audio and MIDI data. These settings, including the processing used to play or generate sounds, are collectively called a sound set. Only one type of sound set can be configured at a time. However, the bather can switch between various sound sets using the remote controller. A sound set is designed as an expression of the bathing state. Thus, it is possible to design sets for a variety of bathing spaces. The sound controller is processed either inside the water heater unit or by an external processor over a network. The generated sounds are played in the bathroom via an audio speaker located in the bathroom or in remote environments via networks.

6.2.2.5 Bathonify Remote Controller

In addition to the displays for water temperature and settings, Bathonify requires additional displays in the bathroom for sound on/off, sound setting changes, volume adjustment, operation information, system state, vital signs, and sound set images.

6.2.3 Software Components

This section describes the software components and the sound design of Bathonify. Figure 6.5 shows a schematic of the Max/MSP. The software operates as a sensing processor for the extraction and processing of sound control information, A/D control of water pressure output, and serial communication control from the ECG measurement unit. Max/MSP configures each sound set as a sub-patch (subprogram), which is then imported by the sound processor. If multiple sound sets are used, the sound processing module can switch between them.

Fig. 6.5

Schematic of the software modules

6.2.3.1 Sensing Module

Max/MSP samples the water pressure sensor output at 200 Hz and divides the data into bathing movement information and breathing component extraction information for processing. There are three types of bathing movement information: Entering/exiting state (Bathing: 1. Not bathing: 0.), amplitude value (level change over fixed time), and excess amplitude trigger (above the amplitude threshold). Moving average processing is used to eliminate the noise from the output voltage of the water pressure sensor. Sensor output characteristics are then translated into water level in the bathtub. The water level value is used to judge whether the bathtub is occupied. The bathing state is output as one of two variables (occupied or not occupied). To make this determination, the average water level when the bath is not occupied is used as the standard water level and a certain value above this level is set as the threshold. If the water level is continuously above the threshold over a fixed period of time, the processor judges the bathing state as occupied. If, however, the water level is below the threshold over a fixed period of time, the processor judges the bathing state as not occupied. The threshold for this system is 20 mm above the standard water level. If the water level exceeds the standard value, the processor activates after a fixed time of 0.8 s. We verified the 20 mm value for the shape of the bathtub used in this system. However, a bathtub with a smaller volume would experience larger water level changes and surface ripples, necessitating the setting of a higher threshold. If the fixed time threshold of 0.8 s is set shorter, the processor makes incorrect judgments on large water surface ripples. On the other hand, if the threshold is set longer, the sound that starts based on the trigger that detects when a person is entering the bath would be awkwardly late. We therefore determined that 0.8 sec is an appropriate time. The amplitude value is set for a time level of 1.5 s. The processor will activate an excess amplitude trigger if the amplitude for 0.8 sec of moving time is above the threshold. However, the trigger only activates during bathing. This threshold is related to how easily sound plays based on the trigger; so it can be adjusted by the user.

The breathing component and bathing movement information are processed simultaneously. Water pressure data are then converted into water level data by applying a low-pass filter with a cutoff frequency of 0.35 Hz. The data are subsequently converted again using the bathtub volume and a breathing curve that shows changes in breathing volume outputted. This breathing curve is then passed to the sound controller. For the heartbeat, the measurement circuit outputs an ECG signal, which is sent to the computer over the RS-232 C interface. The ECG waveforms are output directly to the sound processor, which detects R waves and T waves and outputs height values for each of these waveforms as event signals used for timing control. When the processor detects R waves, it finds the instantaneous heartbeat from the interval between the R waves (time interval between the current R and previous R waves).

6.2.3.2 Sound Controller Module

The sound processor sends the control signal received from the sensing part to the sound set (Max-patch). The previous sound set sent from the control signal can be changed as required using the bathtub remote controller and the mixer can adjust the sound volume before the sound is output to the speaker. The Max-patches (programs) arrange the data input port that receives control information and the sound data output port. They have an integrated input/output interface, which makes it is easy to switch between sound sets.

6.2.4 Sound Design

In our prototype system, we prepared two different sound sets to verify that they could be switched when it was being used.

6.2.4.1 Sound Set 1

We designed Sound Set 1 to be similar to Soundscape (Schafer 1993) with an underlying sound of gentle waves lapping a tropical seashore in the evening. The lapping wave sound is mapped so that its volume can be changed using amplitude values. When the bathwater is stirred strongly, the lapping wave sound increases. Different amplitude values are produced for stirring of the water when the bath is occupied and for when it is not occupied. The sound set is configured so that an occupied bath produces louder sounds for small amplitude values compared to an unoccupied bath. Entering or exiting the bathtub changes the value of a two-state variable. Whenever the state changes, a sound like the crunching of seashells plays. This crunching seashell sound is the same for entering and exiting the bathtub; however, the sound plays longer when the bathtub is being exited. In this way, it is possible to distinguish between entering and exiting of the bathtub by the length of the crunching seashell sound. We also added sound components that only play while a person is bathing. They include a piano sound and a sound synthesized based on an image of the moon. The synthesized sound repeats an underlying bass sequence phrase, to which various alto phrases are randomly selected and added. The tempo and expressive probability of an alto phrase can change based on an amplitude trigger. Like the synthesized sound, various piano phrases are available. Piano phrase selection and occurrence timing are controlled by an amplitude trigger. However, if the amplitude trigger detects a new state while one phrase is playing, a new phrase is not played immediately. This control prevents the occurrence of a cacophony.

6.2.4.2 Sound Set 2

Sound Set 2 is a background music component that expresses the bright and relaxed ambiance of a tropical seashore at noon. With an underlying sound of guitar strings, various synthesized sounds and effects come into play when the pressure changes as the bathwater is stirred. The tempo of the guitar emulates the heartbeat of the bather. Thus, it is possible to be aware of one’s own beating heart. The heartbeat emulator harmonizes completely with the instantaneous heartbeat, so it is possible for the music tempo to change sharply. To control sharp changes, the heartbeat is averaged over five values. During a quiet bathing state, a clear, refreshing synthesized sound plays in harmony with the amplitude of the breathing curve. One objective of this design is to evoke a feeling of relaxation as the sound harmonizes with the deep breathing of the bather.

6.2.5 Bath Remote Control

Figure 6.6a shows an image of a bath remote control operated from a computer that has a touch panel display. Figure 6.6b shows the remote control’s display window. In addition to conventional displays for hot water supply and bath temperature settings, this remote control has various other settings in the display window, such as sound set selection, display of the water level with surface wave information including breathing information, display of ECG data and heartbeat information, and sound volume control for each sound set.

Fig. 6.6

Bathonify remote control a Touch panel display, b Examples of display window examples

6.2.6 Evaluation of Bathonify

In order to verify that the Bathonify system provides an amenity space that bathers enjoy, we conducted a simple evaluation. Thirteen male subjects ranging from 20 to 35 years old were asked to evaluate the bath system one at a time and experience the interactive sound, after which each subject completed a questionnaire. The questionnaire asked the subjects to give their impressions about various aspects of the system on a scale from one to seven: Enjoyable (1-Not enjoyable to 7-Enjoyable), calming (1-Not calming to 7-Calming), and palliative (1-Not palliative to 7-Palliative). Space was also made available for them to freely write comments. We set the room temperature at 28 °C and the hot water temperature at 40 °C (the actual temperature of the water in the bathtub was 38 °C), which are normal bathing conditions. We used Sound Set 1 and left the bathing time up to each subject’s discretion. Figure 6.7 gives the average and standard deviation of the evaluation results.

Fig. 6.7

Result of subjects’ answers

Figure 6.7 shows that the subjects felt the experience was enjoyable and palliative. These positive results suggest that the system is efficacious for bathers. Nine out of the 13 subjects wrote favorable comments about the interactive environment under the enjoyable aspect. Their comments suggest that the system fulfills its objectives. Eight subjects who evaluated the calming aspect at three or below felt a sense of discomfort from the novelty of the experience or that the wave volume was a bit too loud. Other comments related to sound and sound requests can be handled by the system through remote control operations or by creating new sound sets.

During verification of Sound Set 2, one bather observed that the music tempo increased the more deeply he became engaged in the sound interaction. Prior to this, the subjects had only been aware of the music tempo, not their rising heartbeats. The experience of this subject, though, suggests that it is possible for a bather to notice his physiological state through interactive sound information.

The Bathonify system was demonstrated at the IPSJ Interaction 2002 conference in Tokyo, and also exhibited in a home equipment showroom in Osaka for two weeks in February of 2006. It has also won several awards and earned praises as an enjoyable system.

6.2.7 Bathonify Concluding Remarks

This section described the Bathonify system, for which the focus is the provision of a ubiquitous computing, healthcare, and entertainment bath system environment with interactive sonification tuned to bathing motions and vital signs. The system measures the bathing state using sensors inside a hot water heater and an electrode attached to a conventional Japanese bath system. Bathers can take baths in their customary fashion because the sensors are embedded and hidden, and they can select from various sound designs and adjust the sound volume as they like. This system can also be used to express life log data, especially for bathing activities in everyday life.

6.3 TubTouch

6.3.1 System Overview and Design

TubTouch (Sakakibara et al. 2013; Hirai et al. 2013) provides an integrated user interface and several interaction features in the bathtub for the control of various equipment and applications. As illustrated in Fig. 6.8 capacitive touch sensors are attached to the inside edge of the bathtub to enable bathers to interact by simply touching the bathtub. A video projector installed above the bathtub projects virtual buttons and/or a screen for applications over the touch sensors, shown in the picture on the right side of Fig. 6.8. In Japan, standardized bathroom systems are widespread in homes, including houses, condominiums, and apartments. The bathrooms are constructed from unit elements, such as wall, floor, ceiling, bathtub, and are relatively easy to assemble and remove. The space inside the side of the bathtub can be accessed by removing a side panel; resulting in easy installation of capacitive touch sensors, shown in the picture on the left side of Fig. 6.8. The space in the side of the bathtub was designed specifically for additional equipment such as a Jacuzzi. The picture on the left side of Fig. 6.8 also shows several electrodes on the upper inside edge of the bathtub, and a sensor box containing a touch sensor controller board. This arrangement means that TubTouch can be installed as an additional system in any such existing bathroom. In addition, electrodes can be freely installed on the rear side of surfaces, including curved surfaces. Another advantage is the flexibility of the interactive display and its compatibility with conventional household environments.

Fig. 6.8

Tubtouch system overview

Capacitive sensors usually respond to contact with water and are therefore used to measure water levels in tanks. Recent multi-touch input devices tend to be incompatible with wet environments. However, the basic function of a capacitive sensor is to react to the presence of dielectric objects. Since water and the human body have different relative permittivity, TubTouch can indeed be used to detect human touch, even when wet, in response to each sensor signal.

Japanese bathroom systems also have space in the ceiling to install equipment such as ventilators, dryers, mist generators, loudspeakers, and audio units. An access hatch is provided in the bathroom ceiling for easy access to this space; thus, a projector can very easily be installed there.

There are three ways to interact with TubTouch: touching, sliding, and proximity to the edge of the bathtub^{1 , 2}. As mentioned above, the proximity value is measured by reaction to the presence of dielectric objects, such as fingers and hands in this case. Touch detection is a proximity state that can be determined quite simply using a threshold. Sliding motions can be detected by transitions of proximity values from multiple electrodes.

6.3.2 TubTouch Entertainment Applications

TubTouch has several entertainment applications, including control of some bath equipment such as lighting, audio, and TV. In this section, we introduce three entertainment applications that, in particular, provide new experiences during baths.

6.3.2.1 Bathtuboom

Bathtuboom, shown in Fig. 6.9, is a kind of interactive art system. Each colored ball projected onto the edge of the bathtub is a button that is able to activate sound phrases and move light shapes on the top of the bathtub. When a bather touches these buttons simultaneously, the overlapped sound phrases generate music that can be listened to. Bathers, especially children who dislike bathing, can experience some amount of pleasure by touching these balls and listening to the resulting music.

Fig. 6.9

Appearance of bathtuboom

6.3.2.2 Batheremin

Batheremin, shown in Fig. 6.10, is a theremin application. The theremin is a very famous electronic musical instrument that is controlled by the proximity of the two hands via capacitive sensors. We designed a bathtub embedded touch sensor system as a theremin that can be played using two hands.

Fig. 6.10

Appearance of batheremin

6.3.2.3 BathCount

When taking a bath, many Japanese children play a game, while in the tub of water, in which they count from one to a few tens until the end of the bath. Children learn numbers and counting through these experiences with their parent(s) in the bathroom. BathCount, shown in Fig. 6.11, is a kind of support system for this counting experience. When a button is touched, BathCount displays a corresponding number on the bathtub, and speaks the number or plays some sounds. Children using this system can count numbers with or without a parent, resulting in more fun at bath time.

Fig. 6.11

Appearance of bathcount

6.3.3 TubTouch Concluding Remarks

In this section, the TubTouch bathroom system was described. This system enables a bathtub to become an interactive controller using embedded capacitive touch sensors, and a number of new entertainment applications associated with it. The TubTouch system and these applications have been exhibited and demonstrated at several exhibitions and conferences, for instance, at Makers Faire Tokyo 2012 and IPSJ Interaction 2012. The general feedback from people who experimented with the applications has been very positive; they usually state a desire to have the TubTouch system and its applications in their own homes.

We plan to develop middleware for TubTouch, using the TUIO protocol (Kaltenbrunner 2009) to divide it into hardware and software platforms, in order to make applications easier to develop.

6.4 Bathcratch

The sounds that a bathtub makes when rubbed, brushed, or struck are familiar to virtually everyone. We propose using the bathtub as an interface for creating music. To explore this concept, we developed Bathcatch (Hirai et al. 2012), a system that detects the squeaks made when rubbing a bathtub, as well as the sounds made by other such actions, and converts them into musical sounds (see Fig. 6.12)³. By embedding sensors that can detect touch and sounds, the bathtub is virtually converted into a user interface (UI) for a DJ controller. We intend for this to be a new way to make everyday activities more fun.

In this section, we present a system overview and describe the method by which scratching sounds are processed to associate interaction with rhythm tracks. In addition, we describe the feedback received from the public at an exhibition where the Bathcratch system was installed.

Fig. 6.12

Appearance of Bathcratch

6.4.1 System Overview

As shown in the overview in Fig. 6.13, a contact microphone (a piezo sensor) is attached on the inside edge of the bathtub at the point where the right hand of the user would normally be placed. The microphone senses squeaks made when the bathtub is rubbed as solid vibrations in the body of the tub. The sounds are processed by a software called the Squeaking Sound Detector, which handles the rubbing input. For the left hand, capacitive touch sensors are provided, which allow various other inputs to be given to Bathcratch. These embedded sensors represent one novel feature of the system: they are invisible and do not impede everyday cleaning of the bathtub. A video projector installed above the tub projects virtual buttons over the touch sensors on the left side and marks the input area on the right for the contact microphone. Another novelty is the flexibility of the interactive display and its compatibility with the ordinary household environment. The Scratch Music Controller generates scratching phrases according to the squeaks detected and also changes the scratching effects and rhythm tracks in accordance with the touch inputs, as illustrated in Fig. 6.14.

Fig. 6.13

Bathcratch system overview

Fig. 6.14

Internal process of the Bathcratch system

6.4.2 The Bathtub as an Interaction Medium

6.4.2.1 Detecting Squeaks

This system must detect and differentiate between various squeaks and play associated scratch sounds. These squeaks have subtle differences depending on the material of the bathtub and the way it is rubbed, for instance, with different finger angles, rubbing directions, and pressure values. However, these sounds have a fundamental frequency (F0) and specific harmonic structures, as shown in Fig. 6.15. This spectrogram shows the harmonic structure and its continuous characteristic. We confirmed that the same characteristic exist for various bathtub squeaking sounds. The range of F0 is 100-600 Hz.

In addition to squeaks, taps and knocks on a bathtub also produce solid vibrations, although they do not have the same characteristics as squeaks; they have short durations and do not have a distinct harmonic component, as shown in Fig. 6.16. The other sounds with harmonic components in a bathroom are human voices. However, we confirmed that a contact microphone attached to a bathtub filled with water will not detect a human voice. Thus, in order to isolate squeaks, the system must identify signals with a certain continuous harmonic structure and amplitude. However, the current system does not detect a continuous harmonic structure accurately. Therefore, the external object sigmund in the Max/MSP software environment is used to estimate F0 instead.

Fig. 6.15

Spectrogram of a bathtub squeak

Fig. 6.16

Spectrogram of a bathtub knock

6.4.2.2 Using TubTouch and Projection Display on Bathtub

Bathcratch partially utilizes the TubTouch system, which enables switching of rhythm tracks, scratching phrases, and sounds while playing. The embedded contact microphone and capacitive touch sensors are invisible and no changes are made to the surface of the bathtub, as mentioned above. Instead, a video projector installed above the tub projects virtual interactive objects over the touch sensors and indicates a designated rubbing area near the microphone (see Fig. 6.12). Note that the contact microphone (piezo sensor) can be installed anywhere on the edge of the bathtub as the solid vibrations are conducted quite well through the bathtub. The designated rubbing area is only intended as a visual aid to prompt the user to rub the bathtub.

6.4.3 Scratching Music Controller

6.4.3.1 Overview of Scratching Phrase Generation

We implemented the Squeaking Sound Detector and Scratching Music Controller were implemented as a Max/MSP patch, as shown in Fig. 6.17. The checkboxes at the top are toggle switches to control the entire Bathcratch system in order to play specific rhythm tracks and to change the pitch of the scratching sounds. The faders control the volume of each scratching phrase and the master output. The checkboxes in the middle can be used to make scratching phrases, as described in the next section. To the right of these checkboxes is an option to set the tempo for the rhythm tracks and scratching phrases. The current Bathcratch system does not generate scratching sounds synchronized with the actual rubbing motion, but generates phrases synchronized with the tempo of the selected rhythm track. This is because of the latency in detecting squeaking sounds and the difficulty users encounter trying to rub and make squeaking sounds coordinated with the rhythm track. Therefore, we designed the interaction with Bathcratch such that rubbing actions (making squeaking sounds) are first used to prearrange a set of scratching phrases, which can be switched by using the touch controls. Thus, any user can intuitively create DJ scratching sounds with relative ease. We plan to implement synchronized, real-time scratching for experts in the next version of Bathcratch.

Fig. 6.17

Max/MSP patch for Bathcratch

6.4.3.2 Switching Between Scratching Phrases

Five scratching phrases can be arranged freely based on the basis of half notes, quarter notes, eighth notes, sixteenth notes, and triplet notes. These phrases are prearranged with checkbox groups in the middle area of the patch shown in Fig. 6.17, and icons representing each phrase are projected on the top of the touch sensors. Users can switch between the five scratching phrases by touching the projected objects. The five phrases are always played in the tempo of the current rhythm track, and Bathcratch outputs only the phrase selected by controlling the faders for each phrase.

Consequently, even if a user rubs vigorously and quickly, the output phrase is not changed. Furthermore, when a user selects another phrase before the current phrase has completed, a smooth transition is made using crossfading effects.

6.4.3.3 Sound Sources and Effects for Scratching

Currently, this system plays prerecorded scratching sounds for each note in the scratching phrases. It is possible to assign a single sound source and slightly change each note in the phrase and to change the playback speed of the assigned source to create individual effects. By using these functions, the same sound source can be used in a phrase, although each note played is not the same. This reduces the number of sound files and materials necessary and makes it easy to create a phrase with a few sound sources. Even if only one sound file is assigned to all the notes of all phrases, a wide repertoire of phrases can be realized by randomly changing playback speeds. Although individual assignments are performed before playing with this system, the randomizing mode can be controlled in real time using the touch inputs. This method of sound assignment and generating effects makes the phrases seem more natural and nonmechanical. In addition, actual DJs employ a variety of techniques on real turntables and faders, for instance, chirp scratch, forward/backward scratch, and transformer scratch. These functions of Bathcratch can be considered as simplified and modified functions that are carried out using actual turntables and faders.

6.4.3.4 Rhythm Tracks

We prepared a range of background rhythm tracks, such as, OldSchool, Dubstep, JazzyHipHop, and Electronica, for the scratching performances. Users can select a rhythm track by sliding the track selection area over the touch sensors. However, the rhythm track manager always plays all tracks in parallel and only turns up the volume of the selected rhythm track while muting the others.

6.4.4 Demonstrations and Exhibitions

The initial version of Bathcratch with a simple UI can be seen on YouTube (Fig. 6.12). The installation version of Bathcratch (Fig. 6.18) with an improved UI (Fig. 6.19) was exhibited at the 2010 Asia Digital Art Awards at the Fukuoka Asian Art Museum in Fukuoka, Japan, from March 17 to 29, 2011. It was also been exhibited at the National Museum of Emerging Science and Innovation (Miraikan) in Tokyo on October 10, 2011. The UI was changed for this version because it was operated on the sides of the bathtub (see Figs. 6.20 and 6.21)⁴.

Fig. 6.18

Bathcratch installation

Fig. 6.19

User-interface of the Bathcratch installation

Fig. 6.20

Exhibition at the Fukuoka Asian art museum in Fukuoka, Japan

Fig. 6.21

Demonstration at the national museum of emerging science and innovation (Miraikan) in Tokyo, Japan

The UI of the installed version presents a movable gradation square for rubbing on the right edge, as seen when standing to the side of the bathtub. The buttons used to select scratching phrases are along the left side of the square rubbing area. Each button represents a musical note, for instance, a quarter note, an eighth note, a sixteenth note, etc., which represents a fundamental note of a scratching phrase. Users can select and change phrases by touching these buttons. In addition, on the left side, there are effect buttons to change the pitch of phrase notes as well as a sliding selector and a mute button for the rhythm tracks. Each icon of the sliding selector represents the characteristics of the associated rhythm tracks in terms of color and icon design. We provided a wet sponge along with the setup to allow users to wet their fingertips in order to create squeaks when rubbing the bathtub. It was placed near the square area designated for rubbing. A few drawbacks were noticed during this exhibition. One involved the setting of the input gain for the piezo sensor when there is no water in the bathtub. Turning a rhythm track up at high input gain caused misdetection of F0 because of interference with the notes from the rhythm track. This phenomenon does not occur when there is water in the bathtub. Therefore, we think that water acts as an attenuator that blocks surrounding sounds. Another problem was that some users could not understand the difference between the rubbing UI and the touch UI. Most of them did not rub but slid their fingertips lightly on the rubbing area despite the fact that they needed to make squeaks. Fortunately, the exhibition staff explained the operation of Bathcratch and showed users how to use it.

This indicates that we need to improve the UI to more clearly indicate that a rubbing motion that produces squeaks is necessary.

6.4.5 Bathcratch Concluding Remarks

This section described the Bathcratch system, which allows anyone to create DJ scratching sounds by rubbing a bathtub. The system utilizes the squeaks produced by rubbing smooth surfaces, of a bathtub, in this case. This section also described the UIs used for rubbing and touch inputs, which were implemented with a projector, along with the systems for detecting squeaks and controlling the scratching music. Bathcratch has been presented at several exhibitions, where it has been awarded several prizes. Squeaks produced by rubbing smooth surfaces are quite common in everyday life; for instance, when polishing mirrors, windows, bathtubs, and dishes. Therefore, this system can utilize a casual action that occurs in daily life as a means of entertainment. People can control various devices via rubbing motions and squeaks. Moreover, the input functionalities of this system can be increased by including rubbing length and timing as additional parameters. As future work, we plan to simplify the system and analyze the squeaking sounds accurately in terms of the timing. We also plan to analyze the feasibility of including various other aspects of rubbing motion as additional input parameters for a general UI; for example, the number of rubbing fingers, rubbing direction, and the intensity. Finally, we plan to improve Bathcratch’s entertainment functionality, including the music controller, and further explore the concept of entertainment with common objects found in a typical home.

6.5 Related Work

6.5.1 Bathonify

In building Bathonify, reference was drawn from the concept of ambient media (e.g., light, sound, water) as defined by the Tangible Media Group at MIT (Ishii and Ulmer 1997). Ambient media research related to water as a form of visual expression has been used by Dahley et al. in the Water Lamp (Dahley et al. 1998), and by Sugihara et al. in the Dome (Sugihara and Tachi 2000) and in the Water Display (Sugihara and Tachi 2001). Works of art using sound to express water movement include Tangible Sound by Yonezawa (Yonezawa and Mase 2000) and Sound Flakes by Moroi (Moroi 2004). AquaTop display (Koike et al. 2012) uses the surface of the in a bathtub as an interactive visual display.

Sonification research for everyday life activities or life log has also been done. Mynatt et al. attempted to sonify activities at home in the Aware Home project (Mynatt et al. 1998; Tran et al. 2000). Oki et al. attempted something similar using an orgel (Oki et al. 2008) in the OchaHouse project. These sonification research activities tend to not be in the bathroom, but in the living room, bedroom, and kitchen.

6.5.2 TubTouch

Smartskin (Rekimoto 2002; Fukuchi and Rekimoto 2002) is one of the important UI researches that uses capacitive touch sensors into a table and a pad. These are surface computing researches that provide a multi-touch, gesture, proximity and shape input. The TubTouch system references them but is applied to the narrow edges of the bathtub while coping with water.

DiamondTouch (Dietz and Leigh 2001) can identify users. The possibility exists that TubTouch will be used by multiple persons, such as children and a parent, at the same time. This matter can be dealt with by referencing DiamondTouch. Westerman’s system (Westerman 1999) can identify fingers by proximity images from touch sensor arrays. At present, TubTouch is applied to the bathtub’s narrow edges, which restrict arrangements of touch sensor electrodes. Therefore, TubTouch is limited in its ability to recognize finger gestures that are usually used in surface computing.

TubTouch can be applied to not only flat surfaces, but also curved surfaces. There are several research efforts related to curved surfaces computing, for instance, Sphere (Benko et al. 2008) and a dome display by Benko and Wilson (2010). These systems also use video projectors to display on curved surfaces. However, they use infrared (IR) to detect touches.

Touché (Sato et al. 2012) can not only detect a touch, but also recognize how the touch was done. This system uses the capacitive sensing technique with swept frequency and support vector machine to classify the touch context. In comparison with normal capacitive touch sensing techniques including TubTouch, this recognition process needs some latency and a high performance CPU. TubTouch will reference Touché in the future for various interaction designs.

The AquaTop display (Koike et al. 2012), previously mentioned in work related to Bathonify, is also a surface computing system that does not use the capacitive touch sensing technique. This system uses a depth camera to detect some finger and hand gestures at the surface of the water in a bathtub. In order to display on the water surface, a video projector is used, same as with TubTouch.

6.5.3 Bathcratch

Considerable research has been conducted on music systems and UIs for DJ controllers, including inputs for scratching. For example there are experimental turntables and wearable UIs such as the DJammer (Slayden et al. 2005), Music-Glove (Hayafuchi and Suzuki 2008) and Wearable DJ System (Tomibayashi et al. 2009) that allow users to Air-DJ and scratch. Mixxx (Andersen 2003) uses ARToolKit to implement an augmented reality turntable that can play various sounds. D’Groove (Beamish et al. 2003) has a turntable with force feedback as well as a DJ mixer that allows users to practice the fundamental techniques of DJing. Hansen uses the Reactable as an UI for DJ scratching (Hansen et al. 2007; Hansen and Alonso 2008). Fukuchi’s system uses a capacitive multi-touch surface and allows multi-track scratching (Fukuchi 2007). Another turntable controller that includes commercial products for scratching is described in detail in Hansen’s doctoral thesis Hansen (2010).

In addition, some research has been conducted on utilizing acoustic sensing in a UI. Scratch Input (Harrison and Hudson 2008) to detects scratching sounds and the associated finger motions using a piezo microphone attached to a wall, table, etc. Stane (Murray-Smith et al. 2008) attempted to the detection of vibrations when the surface of a small device with built-in piezo sensors is scratched. The device also used various input patterns that depended on the vibration length. Skinput (Harrison et al. 2010) uses sounds and machine learning to implement a UI. The system uses the human body itself as the UI by recognizing finger taps through vibrations transmitted along the skin surface using a piezo film rolled around the upper arm. Lopes’s system (Lopes et al. 2011) uses the sounds of finger, knuckle, fingernail and punch touches, in order to expand the input language of surface interaction.

6.6 Discussion

Japan has a unique bathing culture. A lot of people feel that bathrooms are amenity spaces for refreshing the mind and relaxing. Half-body bathing is typical of this. Hence, general bath modules in Japan have a feature to expand various functionalities with optional equipment; for instance, ceiling speakers for listening to music, ceiling illuminations with spotlights for room effects, and mist generators for beauty and fine skin. People read books and listen to music while in half-body bathing or bathing with mist generators, which can also make sauna baths. In spite of these advanced bathroom situations and environments in Japan, it is important to note that the systems introduced in this chapter can make bathing a more entertaining experience. The common grounds of these systems are the embedded sensors and the provision of interactivity with entertainment while in the bathroom. The embedded sensors have two benefits. The first is that sensors do not hinder the normal cleaning of the bathroom and/or bathtub. The second is that it is a smart environment with many kinds of software applications, including entertainment for various persons, from children to elderly people. This embedded and smart environment realizes one of the concepts of the Ubiquitous Computing Environment (Weiser 1991). Bathonify uses the built-in water pressure sensor of the external water heater, and an ECG sensor. Fully-automatic water heaters are already widespread in Japan and all have a water pressure sensor. The ECG sensor unit also exists as an extension unit for some external water heaters. Thus, the practicability of this system does not depend on hardware, but rather on the software and the network to carry out signal processing, play sounds, and download additional sound sets. TubTouch and Bathcratch need to embed touch sensors and a piezo sensor as a contact microphone into the inside edge of the bathtub. As stated above, most bathtubs in Japan have a removable side panel. Optional equipment, such as a whirlpool and an ECG sensor unit, can be installed inside the bathtub at the side. The picture on the left side of Fig. 6.8 shows a bathtub with its side panel removed, installed capacitive touch sensor unit, and red lines as sensor electrodes. Piezo sensors can also be easily installed in this space. Thus, it is already possible to install these sensors in existing bathtubs. The most difficult part of the system may be installation of the video projector in the ceiling. However, pico-projectors are presently undergoing remarkable development and a waterproof version that can easily be installed in the ceiling will be released soon. Consequently, these systems may soon be released as practical entertainment systems.

The systems described in this chapter will bring interactivity while bathing and utilize a bather’s actions and/or vital signs. Bathers need active interaction to use TubTouch and its applications and Bathcratch. These applications facilitate extraordinary bathing activities and entertainment, so it is envisioned that they will be used by bathers who would like to have fun. Bathonify, on the other hand, can be used with both active interaction and passive interaction. Some bathers may have fun with the interactive sonification of some sound sets using active motions, whereas others may not pay extra attention to the interactive sounds but hear the sounds produced by their vital signs passively. Bathers who dislike sounds while bathing can have the sounds conveyed to another place, such as the kitchen in the home or a remote location where a relative is, while experiencing silence from the system in the actual bathroom. Hence, Bathonify is useful both explicitly and implicitly.

These novel interaction and entertainment systems, which can be used actively or passively, explicitly or implicitly, improve QoL with bathers’ selections.

6.7 Conclusion

This chapter described three interactive and smart bathroom systems that comprise aspects of ubiquitous computing and entertainment systems.

The first system, Bathonify, reflects the sounds made by a bather’s movements and vital signs, such as ECG and breathing. This system can be seen as a kind of interactive sonification system to reflect a bathing life log. Bathers can take their baths as usual because the sensors are embedded in the system. Further, they can monitor their own vital signs and transmit bathing state and other information to another place.

The second system, TubTouch, is a controller platform that transforms a conventional bathtub into an interactive controller using embedded capacitive touch sensors and a video projector. This system provides a surface computing platform for the bathtub that can be used as an integrated controller for various conventional bath equipment, such as water heaters, TVs, audio equipment, Jacuzzis, dryers, and mist generators. In addition, this system can be used to provide novel applications in the bathroom, such as Bathtuboom, Batheremin, and BathCount, as discussed above.

The third system, Bathcratch, is a DJ scratching application that uses the squeaks produced by rubbing the bathtub to provide entertainment. This system uses an embedded piezo sensor inside the bathtub to detect squeaks, and partially utilizes the TubTouch system.

We also introduced works related to the above systems and discussed them from the point of view of practicability, common grounds, and interactivity.

These smart, interactive entertainment systems and technologies for bathrooms will change and improve QoL in the future.