Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Many people nowadays are living apart from their friends and families and their communications are limited to voice or video calls and sending text messages with their mobile phones. Although these communication methods are sufficient for keeping in touch and sharing information about their everyday life, touching or kissing our loved ones over the Internet remains to be a challenge yet to be addressed by most communication systems. This is perhaps due to a lack of understanding of the significance of physical interaction in improving relationship qualities or a reluctance to approach such a sensitive or private topic.

Making regular physical contact is essential for maintaining closeness and intimacy in human relationships, be it romantic or familial. Haptic communication is very effective in conveying one’s feelings and emotions as well as evoking a sense of presence in a remote environment [10]. In particular, kissing is one of the most natural and universal expressions of love and affection. In many cultures, kissing is also a common form of greeting and goodbye. Studies have shown that a higher frequency of romantic kissing between couples increases romantic satisfaction and reduces stress level [3]. Family members, especially parents and children, also kiss each other to express their love and care. Kissing is the most direct and effective way of sharing an intimate moment and internet communications should allow people to connect to each other through this form of interaction.

Kissenger is developed with the aim of enabling users to kiss each other remotely through a haptic interface that replicates real kissing sensations. This is an evolution of our previous work [19], which was a PC-controlled device. The latest development consists of a plugin device that connects to a mobile phone’s audio jack and a mobile application that connects to the device and sends real-time data stream over the Internet. A usage scenario is shown in Fig. 1, a little girl from London kisses her grandmother in Tokyo remotely over the Internet through a mediating device while talking on their mobile phones as if they are in the same physical location. Accurate haptic stimulations are produced by the positional changes of an array of linear actuators in contact with the human lips. Bilateral force control is used such that both users feel the reflections of their own forces as well as the forces from each other. Force sensors are placed on top of the linear actuators to measure the output forces and provide feedback to the local force controllers.

Fig. 1.
figure 1

Concept of Kissenger. A little girl from London sends a kiss to her grandmother in Tokyo over the Internet

Full size image

Finally, we will also explore ways to engage other sensory modalities such as digitally stimulating the senses of taste and smell [18] to provide a multisensory kissing experience.

2 Related Work

Researchers and the industry have been actively developing systems, frameworks and architectures for integrating haptics in various applications, e.g. augmented reality, teleoperation, task simulation. Universal haptic interfaces, such as Novint Falcon [15] and HapticMaster [22], offer some out-of-the-box options to implement haptic control systems without a tedious development cycle. However, these readily available systems are not made to be used with mobile phones, which pose several limitations on the form factor, bandwidth and computational resources. Haptic solutions for mobile devices are mostly restricted to vibrotactile stimulation rather than force feedback. It’s necessary to design a portable task-specific system suitable for physical interactions (such as kissing) using mobile devices.

2.1 Kiss Communication

Previous works that explored the concept of telecommunicating kiss did not focus on transmitting the dynamics of forces. The Kiss Communicator by IDEO lab [2] is a concept prototype that records the breath pattern of the sender as he blows into it and plays back in the form of an animated light sequence. In [6], the moisture level on the kissing device is measured and transmitted to the receiver by varying the wetness of a motorized sponge to represent a kiss. The Kiss Transmission Device [20] from the Kajimoto Laboratory is a real-time bilateral system that synchronizes the turning angle of a straw as the user swirls it around using his tongue.

2.2 Admittance and Impedance Controlled Systems

Differentiating the system structures of haptic devices is important at the outset of the design process. Two general classes of systems can be identified depending on the mechanical inputs and outputs of the haptic devices - admittance controlled systems and impedance controlled systems [8]. Admittance controlled systems measure force as inputs and generate motion outputs (position, velocity, acceleration) [7, 23]. They are often found in braille devices and haptic displays using actuator arrays. An example is Project FEELEX, which simulates haptic sensations and the texture of an object’s surface using a linear actuator array [7]. In this system, the positions of the actuators are binary (up or down).

Impedance control systems are the reverse as they accept motion inputs and produce force outputs [4]. Both approaches could be implemented as open-loop or closed-loop if force or velocity sensors are used to provide feedback to the controllers.

2.3 Bilateral Control Methods

Designing a real-time bilateral force control to simulate the dynamic forces on the lips of the users is essential for a haptic interface capable of conveying a realistic kissing experience. Stability and transparency are two important factors used to measure the performance of a bilateral control system [9]. Stability depends on how well the controller counters time delay and package losses in the communication channel, which are major disturbances to the system. Transparency measures how closely the two devices (master and slave) follow the motions and positions of each other. In an ideal system with perfect transparency, the human operator should feel that they are directly touching the remote object without a mediating device [17]. Complete transparency cannot be achieved if the system is unstable. It is necessary for the controller to reduce the effects of time delay.

Many bilateral tele-manipulation systems are developed for humans to explore a remote environment or to interact with remote objects via a robotic manipulator [24, 25]. Human to human haptic communication over a network is investigated in some tele-handshake systems and they demonstrate different force and position control algorithms with disturbance compensation [12, 16, 21].

Solutions such as the scattering theory control by Anderson [1] and the wave variables control by Niemeyer [14]. Both approaches maintain stability by providing corrective actions to preserve passivity of the system. They convert time delay, which is modelled as a non-passive block, into a passive block to maintain passivity. The bilateral controller used in the remote handshake system described in [11] is based on scattering and wave variables control for stabilization and handling communication latency.

A drawback of the scattering approach or the wave variables control is that they deteriorate tracking performance. It is shown that using a local PID controller can attenuate this effect. When using a scattering matrix or wave filter for stabilization, A PID controller implemented at the slave side can improve the tracking performance of the system [26]. Using a PID controller alone on each side of the bilateral system also preserves the passivity of the system [5].

3 Design Considerations

Several factors are considered in designing the control system and hardware of the haptic communication device for mobile phones, including the mode of haptic stimulation, portability, communication delay, bandwidth and the stimulation of other sensory modalities such as smell.

Vibrotactile stimulation is commonly used as the mode of stimulation in many haptic systems as vibration motors are lightweight, cheap and easy to control. The vibration frequency and magnitude are varied to produce different textures and tactile sensations. However, the dynamics of the movements and pressure felt by the human lips during kissing cannot be accurately reproduced by vibrations alone. The mechanoreceptors on our lips respond to the changes in skin strain and deformation caused by external forces exerting on them. The haptic device needs to generate a series of localized forces in order to capture the haptic sensations during kissing. We designed an array of linear actuators positioned evenly across the lips to generate normal forces on the skin surface. The same number of force sensors resistors are placed on top of the actuators to measure the contact forces between the human user and the haptic interface. Although linear actuators with position encoders or embedded force sensors are commercially available, most products are too big for our application. Hence we use force sensor resistors in our system to provide force feedback.

In a field study conducted with the previous version of Kissenger, couples responded that it would be more convenient if they could use Kissenger while talking to their partners on the phone [19]. For a device designed for mobile phones, the challenge is to balance its functionalities with its size and weight in order to have a portable device. Small and lightweight components that consumes minimal power need to be chosen for the hardware. The study also mentions that some people feel embarrassed to kiss a machine in public as they are concerned that other people might find them weird. Thus the device should be designed in such a way that users feel comfortable using it in public. We believe that designing it as a plug-in gadget for mobile phones instead of a standalone device reduces the awkwardness of using it in public.

As discussed in the previous section, time delay present in the communication channel should be taken into consideration in designing the system control. Since mobile devices have lower memory and computational resources compared to computers, the data transmission speed between the haptic interface and mobile phones using audio jack is also relatively limited, the controller should accommodate a low sampling frequency with minimal memory usage and processing requirements.

Smell is another important aspect that directly affects our emotional responses to a kiss. It is well known that our sense of smell affects emotions more efficiently than any other sense. When two people are kissing, they are in close proximity enough to smell the body odor of each other. Experiments have shown that we are able to detect a class of genes present in body odor called the MHCs, which determines our sexual preference and compatibility to a person [23]. Hence, emitting the partner’s body odor during kissing does not only create a stronger sense of physical presence but also strengthens our affection to that partner. Similarly, taste stimulations could also be integrated for a full multisensory kissing experience.

4 System Architecture

Considering the different aspects described in the previous section, we designed the Kissenger haptic interface as a small portable device that can easily plug into the audio jack of any mobile phone. Figure 2 shows a block diagram of the system architecture.

Fig. 2.
figure 2

System architecture of Kissenger consists of human users, a haptic interface with bilateral force control and a wireless Internet communication interface

Full size image

4.1 System Control

During operation, a human user on each side of the system interacts with the haptic interface and exerts a varying force on the lip-like surface of the interface, represented by F ah and F bh in Fig. 2. The controller changes the positions of the linear actuators to reflect the forces transmitted from the partner and the user’s own forces. The forces felt by the user’s lips are the contact forces (F a and F b in Fig. 2) between the lips and the haptic device measured by the force sensors on the surface of the device. In a perfectly transparent bilateral control system, the law of action and reaction must be realized hence the contact forces acting on both users should be equal at all times. In other words, the net force should be zero at all times. The positions of the actuators should also follow the positions of the partner user. Consider two users in the system, User A and User B, and factor in the communication time delay, the objectives of the controller can be expressed as follows:

$$ F_{a} (t) - F_{b} \left( {t - \tau } \right) = 0, \,x_{a} (t) + x_{b} \left( {t - \tau } \right) = 0. $$
(1)

where F a , F b, are the contact forces between the user and the haptic device, x a , x b are the positions of the actuators relative to the origin for User and User B respectively. τ is the time delay of the communication channel.

The positions of the actuators are modelled as the lip surface of the remote human user and should reflect the forces of both users. The position of an actuator could simply relate to the net sum of the two contact forces by a proportional gain, as given by Eq. (2).

$$ x_{a} (t) = K_{s} (F_{a} (t) - F_{b} (t - \tau )). $$
(2)

Different from other teleoperation systems with a master-slave configuration, in which human operators control a remote environment or object through a haptic interface, there are two active user inputs in this system and there is no distinction between a slave and a master.

A bilateral force feedback control is used in the system. Figure 3 shows the control block diagram of the system. A local PID controller is implemented on each side of the system to control the actuators using force data from both users. F ah and F bh are the forces exerted by users, u a and u b are the input commands to the actuators, y a and y b are the contact forces measured by the force sensors, and e a and e b are the errors between the force outputs from both sides. The system is closed-loop admittance controlled as it measures force as input and generate a positional output. Since the actuators do not have position encoders, the output forces are measured by force sensors and fed into the controller to close the control loop.

Fig. 3.
figure 3

Control block diagram for a bilateral force feedback haptic telecommunication system

Full size image

The output of the PID controller is denoted in Eq. (3). The system is discrete with a certain sampling frequency, as denoted by Eq. (4).

$$ u_{a} (t) = K_{p} \left\{ {e_{a} (t) + \frac{1}{{T_{i} }}\int_{0}^{t} {e_{a} \left( \tau \right)d\tau } + T_{d} \frac{{de_{a} (t)}}{dt}} \right\}. $$
(3)
$$ \mathop \to \limits^{\Delta } K_{p} \left\{ {e_{a} \left( {\varDelta t} \right) + \frac{1}{{T_{i} }}\sum\nolimits_{k = 0}^{t} {e_{a} (k\varDelta t)\varDelta t} + T_{d} \frac{{e_{a} (\varDelta t)}}{\varDelta t}} \right\}. $$
(4)

where

$$ e_{a} = y_{b} - y_{a} ,\quad e_{b} = y_{a} - y_{b} . $$
(5)

and \( K_{p} \) is the proportional gain, \( T_{i} \) is the integral time and \( T_{d} \) is the derivative time constant.

The constants are tuned depending on the linear properties of the force sensor derived from calibration, the stroke length of the linear actuator as well as the perceived stiffness of the haptic interface. Upper and lower limits of the actuator position are imposed to ensure safe operation.

4.2 Data Transmission

Force data is continuously transmitted between the two haptic interfaces via the Internet. As shown in Fig. 2, there are two stages in the data transmission channel - (1) audio signal transmission between the haptic interface and the mobile phone and (2) internet transmission between the two mobile phones.

Force sensor resistors are read by the microcontroller’s analog inputs as 10-bit data. In order to maximize the speed and minimize error during data transmission, the data is compressed into 8 bits at the expense of resolution. Data is transmitted between the microcontroller and the mobile phone over audio signals using the Frequency Shift Keying (FSK) technique. The byte-sized force data is modulated to 4900 Hz for a low bit and 7350 Hz for a high bit and sent to the connected mobile phone through the microphone channel of the audio line.

The mobile phone demodulates the audio signal carrying the force data into digital bytes and sends it to the partner’s phone over the Internet. When the other phone receives the force data, it processes the data and modulates it using the same FSK settings over an audio signal and sends to its haptic device through the left/right audio channel.

Latency occurs in both stages of the data transmission process. In the first stage (microcontroller to mobile phone), latency is consistent and controllable whereas in the second case (mobile phone to mobile phone) it depends on the quality of the internet connection of used by the devices. The total time delay is the measured as the time taken to send 1 byte from one haptic device to the other.

5 Implementation

We made a prototype of Kissenger for iOS devices (shown in Fig. 4), including the hardware of the haptic interface and an iOS mobile application.

Fig. 4.
figure 4

Prototype of Kissenger. Users exchange kisses while having a video chat

Full size image

The model of the haptic device is designed and made using a 3D printer with hard PLA material and soft rubber material. Arduino is used as the controller in the haptic device. RGB LEDs are used to provide users a visual feedback and to express emotions through colour. The colour of the LEDs changes according to the duration and intensity of the kiss.

5.1 Hardware

The haptic device consists of linear stepper motors, force sensor resistors and an Arduino microcontroller. The stroke, output force and power consumption of the actuators are some important parameters to consider when selecting linear actuators. There are several types of actuators that generate linear motion. Shape-memory alloy actuators produce weak forces and the output forces are difficult to control. Piezoelectric actuators normally have high driving voltages in the rage of 60 V–230 V, hence not suitable for battery operated devices. High precision miniature stepper motors are chosen for their size, power, output force/torque and controllability. A combination of lead screw and nut converts the rotary motion of a stepper motor into linear motion, making them into linear actuators. Precise positions changes can be controlled using microstepping.

Stepper motors consume the most power when stationary, hence they tend to run hot during standby. In order to avoid overheating, the motor drivers cut off the current to the motors after a period of inactivity.

Force sensor resistors (FSRs) are chosen to measure the output force for their flexibility, low cost, thinness and lightness. Although they are more prone to drift and are generally less accurate than load cells or strain gauges, the advantages of FSRs in size and power outweigh these drawbacks especially for a small size mobile device.

A 4-pin TRRS audio connector is used to connect the hardware device to iPhone. A FSK circuit is built and a FSK modem is implemented on both Arduino and iPhone for data transmission between the two devices. The baud rate of FSK transmission on Arduino obtained from empirical testing is about 100 bps.

5.2 Mobile Application

A mobile application is developed for iOS devices. Figure 5 shows two screenshots of the application.

Fig. 5.
figure 5

Screenshots of the Kissenger iOS app. Left: Users can choose a friend and start a video chat. Right: During the video chat, users can kiss each other using the device

Full size image

It detects and connects to the hardware device when one is plugged in. Users can log in with Facebook, search and add their friends who are also using Kissenger. When a user starts a video chat with a friend, the application starts to send and receive data from the haptic device. Real-time force data is transmitted to the partner via the Internet using the Pubnub data streaming service. Users can also choose to change the LED colours of their partner’s Kissenger device to convey different moods. A disadvantage of connecting to the audio jack of the phone is that headphones cannot be used during conversations.

6 User Scenarios

Kissenger can be used for both one-to-one communication and one-to-many communication where many Kissenger devices are connected to the same network.

Long distance lovers often communicate with their partners through video chats. With Kissenger, they can enhance this experience by kissing each other while looking at their faces and hearing their voices from the mobile phones. Parents can also use Kissenger to give their children a kiss on the cheek when they are away at work, as shown in Fig. 6.

Fig. 6.
figure 6

Usage scenario of Kissenger between a parent and a daughter

Full size image

Apart from personal communications, Kissenger can also serve as a creative gadget for advertising or marketing celebrities. Pop idols can use Kissengers to interact with their fans by sending them a kiss on stage. Multiple Kissenger devices are connected to the same network and one device is selected to be the sender while others are receivers. Each receiver’s device is actuated at the same time when the sender is sending a kiss.

7 Conclusion

We present a kiss communication interface for mobile phones that enables long distance families and friends to remotely kiss each other over the internet. This encourages physical interaction essential to maintain intimacy in long distance relationships.

7.1 Future Work

In the current prototype, the haptic device is attached to the bottom of a mobile phone which has limited space to embed the circuit, controller and multiple pairs of actuators, motor drivers and sensors. We redesigned the model as a mobile phone case (shown in Fig. 7) to make more space at the back of the phone so that it can accommodate more sensors and actuators. With the added space for hardware, the device still keeps its portability with improved aesthetics and functionality since many people use mobile phone cases as protection or decoration.

Fig. 7.
figure 7

3D models of the Kissenger haptic interface in the form of a mobile phone case with additional actuators and sensors

Full size image

The 3D design in Fig. 7 shows the latest model is capable of housing 6 pairs of actuators and sensors, 3 for each lower lip and upper lip. We believe that this new design would generate a more realistic kissing sensation. A more flexible and softer material close to the human skin will be used for the lip part.

Currently the system only implements a single degree of freedom force control. We will explore multiple degrees of freedom and implement more realistic kissing dynamics such as tongue movements. A fuzzy logic PD controller [13] will be experimented considering the different modes of operation, such as when one side is not in contact with the user or in free motion.

We will also investigate ways to incorporate the scent communication device, Scentee, to emit the perfume or a distinctive scent associated with the user’s partner to create a multisensory experience. Lastly, other factors such as temperature and moisture will also be considered in our design.