Keywords

1 Introduction

In the real world, our senses are constantly exposed to stimuli from multiple sensory modalities (visual, auditory, vestibular, olfactory, taste, touch, etc.), and although the process is not exactly understood, we can integrate/process this multisensory information in our acquisition of knowledge of objects in our environment [1]. Although a complete overview of the senses is beyond the scope of this paper, an excellent overview of the senses from a variety of perspectives is provided by [2]. The senses interact with one another and alter each other’s processing and ultimately our overall perception. Most scientific research suggests that the more modalities that are integrated into a virtual representation of our environment, the greater the sense of presence or immersion in that space (see e.g., [3, 4]). Additional modalities can reinforce existing information, or provide additional information that cannot be obtained by a single modality alone. For instance, although vision tends to dominate, auditory information can tell us what is behind a door, or behind or inside our bodies. With respect to learning, training and memory, most studies acknowledge that multiple sensory inputs result in improved processing and retention [5]. Moreover, when there is a deficit in one modality, another may be substituted: persons with hearing impairment sometimes perceive sound with touch, for example [6]. Tactile substitutions for vision are common (e.g. Braille), and “even a poor resolution sensory substitution system can provide the information necessary for the perception of complex images” [7, 8]. For a thorough overview of sensory substitution see [9].

Designers and developers of immersive 3D virtual environments including virtual simulations and serious games, typically aim to faithfully recreate real-world scenarios. In fact, it has been suggested that “achieving multisensory digital experiences is the holy grail of human-technology interaction” in general [10]. However, traditional emphasis is placed on recreating the visual, and (perhaps to a lesser extent), auditory scenes, while ignoring the other senses (including touch, smell, and taste) despite their importance in the real world [11]. Of course, simulating the sense of touch, smell, and taste is not trivial and presents many technological challenges and issues. Recently, there has been a large effort made on simulating the sense of touch (e.g., haptics) and this effort has been accelerated with the availability of consumer-level haptic devices.

Aside from some limited application in virtual environments, the sense of smell has been underused in human-computer interfaces, compared to the use of other senses [12] in multimodal HCI [13]. Studies regarding the role of sensory stimulation in virtual environments has highlighted olfactory stimulation as a potentially powerful yet underutilized therapeutic tool [14]. When incorporated into a virtual environment, the sense of smell (olfaction) may be useful in the treatment of post-traumatic stress disorder (PTSD), a mental health condition triggered by a terrifying event, either experiencing or witnessing it [14]. Furthermore, incorporating pleasant and congruent ambient scents into a virtual environment can lead to enhanced sensory stimulation, which in turn, can directly (and indirectly through ease of imagination) influence affective and behavioral reactions [10]. Finally, incorporating olfactory technologies into virtual environments has shown to be safe and effective for targeting several aspects of psychological and physical health such as anxiety, stress, and pain [15]. In addition, olfactory stimuli at the user interface (UI) may be beneficial in learning, including users’ mood enhancement and stress reduction [12], shifting the learner’s attention to focus on a particular event happening at the UI [13].

There are different techniques used for generating smells in olfactory human-computer interfaces. One of them works by spraying odors into the air [16], used by commercial off-the shelf air fresheners used in households and in other environments, as well as in computer-controlled olfactory interfaces, for example [17,18,19]. These and other projects cited elsewhere use microcontroller boards for controlling and activating sprays. A microcontroller board contains a microcontroller (a small computer on a chip) and extra electronic components that facilitate prototyping of physical computing applications using sensors [20]. Microcontroller boards are effective for controlling actuators such as motors and fans. [21] successfully tested a portable device that generated an odor to enhance user experiences in museums, containing a microcontroller board and two small fans.

However, very few reported olfactory display applications implement an efficient and carefully controlled way to diffuse scents in the air. For instance, [22] used fuzzy logic (it models human logic with intermediate possibilities between digital values YES and NO [23]), a motion sensor and a microphone connected to a microcontroller board for controlling an air freshener, activating it only when it was needed, as an olfactory stimulus to enhance user experiences in large venues. The fuzzy logic analyzed the noise levels and the number of people present in a venue to diffuse a pleasant odor accordingly.

2 Objective

Given the importance of olfaction within the human-computer interface and immersive environments such as virtual simulations and serious games, we have recently begun investigating the use of olfactory stimulation in virtual learning environments. Our ultimate goal is to develop effective virtual learning environments that maximize learning (knowledge acquisition and retention), and we believe this includes an environment where multiple senses (beyond audio and visual) are stimulated. In this paper, we describe the development of a simple, cost effective, and efficient olfactory interface that uses “off the shelf” equipment to provide olfactory stimulation within a typical serious gaming scenario (e.g., a learner seated in front of a computer display playing a math serious game). Inspired by the work of [24], the olfactory interface incorporates fuzzy logic running on an Arduino Uno (a very easy to use microcontroller board, useful in physical computing prototyping [25]), adjusting the amount of scent that it emits based on the distance between the learner (player) and the display (measured by an ultrasonic sensor), and controls a fan placed behind the scent dispenser (an off-the-shelf air freshener) to diffuse an odor. With the limited work focusing on olfactory displays and the growing interest in their use, the work presented here illustrates how an olfactory display can be developed and incorporated into a virtual learning environment in a simple and cost effective manner.

We decided to use fuzzy logic to control our olfactory interface’s fan speed and to activate an air freshener, because when we tested the interface’s ultrasonic sensor we found that when we directly mapped the distance between the player and the olfactory interface the fan speed was done in very small and linear increments, and participants who perceived the smell diffused by the fan did not notice any significant difference. After observing the same effect in a previous study about simulating physical wind in a racing game [26], we noticed in that study that direct mapping of a virtual car velocity to fan speed was done in very small and smooth increments, and participants who tested the fan system in that study did not notice small changes in wind speed. By applying fuzzy logic to control the fan speed in our olfactory interface, the fuzzy logic renders an effective fan speed control (and hence diffusing the smell to the user more effectively) with no perceived delay in the smell diffusion. This in turn could support a satisfactory player’s olfactory experience. The next section provides a brief introduction to fuzzy logic and our fuzzy logic system.

3 What is Fuzzy Logic?

Fuzzy logic is an extension of classic Boolean logic (defining and using true/false values), uses the interval of real numbers between 0 (false) and 1 (true) to describe intelligent reasoning (e.g. 0.89 could mean it is “almost true”), taking into account uncertainties or inaccuracies dealing with imprecise logic values [23]. This has a gradual transition from falsehood to truth. Fuzzy logic is a technique applied in artificial intelligence (AI) to model human logical reasoning with imprecise statements in computer systems such as “it feels really windy outside” where we don’t have a precise and discrete wind value that represents “very windy”. It is based on human reasoning and decision making using a range of possibilities between YES and NO, such as:

  • CERTAINLY YES

  • POSSIBLY YES

  • CANNOT SAY

  • POSSIBLY NO

  • CERTAINLY NO

Those possibilities from a fuzzy logic system are expressed verbally so people can better understand them, and used to represent a more “human-like” output from a system, and not a linear response to the user.

Fuzzy logic can be used to control devices such as motors and consumer products, such as washing machines. This technique may not give accurate reasoning, but acceptable reasoning, e.g. how a washing machine should properly wash dirty but heavy loads of clothes, so it will determine the right amount of water and temperature needed to wash those clothes. Fuzzy logic deals with uncertainty. For example, the speed of a fan can be controlled according to uncertain input values (e.g. distance, or temperature, or humidity changes) from the environment. Thus, it is often necessary to use sensors for measuring values from the environment and use those as input values in fuzzy logic.

In a fuzzy logic system, it is necessary to define fuzzy sets and rules to deal with fuzzy values. Fuzzy rules are conditionals coded in the form of if-then decisions and language functions, such as minimum or maximum value calculations that define behaviours and degrees of membership in a fuzzy system. For example, a fuzzy rule can be: “If it’s forecasted 40% to 50% chance of rain this morning and the barometer is measuring a lower pressure, I will bring an umbrella to work”.

A fuzzy set is an extension of classical crisp sets and are based on the fuzzy set theory. A fuzzy set can be defined as “any set that allows its members to have different grades of membership (membership function) in the interval [0, 1]” [27]. Fuzzy rules define degrees of membership to fuzzy sets. In our previous example, we define two fuzzy sets, one can be composed of rain chance and the other one will be barometric pressure values. Both fuzzy sets can have subsets (membership functions) such as “not likely”, “likely”, and “very likely”. The universe of discourse is the possible range of all values for an input to a fuzzy system (e.g. from 0.0 to 100.0, or [0.0, 100.0]). The crossover point of a fuzzy set is an element at which its membership function is 0.5 [30].

In order to implement fuzzy logic to a real-world application, it must follow three main steps [30] (see Fig. 1):

  1. 1.

    Fuzzification: convert input (crisp) data into fuzzy data and define membership functions and fuzzy sets.

  2. 2.

    Fuzzy inference process: compute fuzzy rules with fuzzy sets to derive the fuzzy output.

  3. 3.

    Defuzzification: convert a fuzzified output obtained from the inference computations into a crisp value with respect to fuzzy sets and their subsets and fuzzy rules.

Fig. 1.
figure 1

The fuzzy logic process.

Full size image

As Fig. 1 shows, the fuzzy logic system inputs crisp values coming from a sensor. The system outputs are crisp values, which will be used to activate an actuator such as a motor, a fan, etc.

3.1 Our Fuzzy Logic System

Since fuzzy logic deals with imprecise levels of truth simulating human reasoning, we defined using preliminary iterative testing with our olfactory interface three fuzzy sets: i) Close (20–40 cm), ii) Near (35–60 cm) and iii) Far (55–80 cm), for the measured distance using an ultrasonic sensor. Figure 2 shows the fuzzy sets. Those were crisp input variables used in our fuzzy logic system that ran on the Arduino Uno microcontroller board.

Fig. 2.
figure 2

Our three fuzzy input sets.

Full size image

As Fig. 2 shows, the universe of discourse for the input values is [20, 80]. Our olfactory interface does not take into account the distance between 0 and 20 cms, since at that distance the user may be too close to the screen and the air freshener, receiving a huge amount of odor on the face, may negatively affect the player experience. Further testing is required to confirm this. In addition, the olfactory interface does not activate when the distance between the interface and the player exceeds 80 cms, meaning that the player is very far away from the computer, or not seated in front of the computer at all.

Our fuzzy logic rules are as follows:

  • IF the player is 20 to 40 cms away from the computer screen THEN rotate the fan at low speed accordingly AND activate the air freshener.

  • IF the player is 35 to 60 cms away from the computer screen THEN rotate the fan at medium speed accordingly AND activate the air freshener.

  • IF the player is 55 to 80 cms away from the computer screen THEN rotate the fan at fast speed accordingly AND activate the air freshener.

Our fuzzy logic generated crisp output values (integer numbers) that fell between 0 (fan stops) and 255 (fan running at full speed). This is because the 8-bit pulse-width modulation (PWM) output function used by the Arduino microcontroller board works with discrete values from 0 to 255. PWM is a technique that can be used to control the spin of motors by setting up the width of pulses (square waves) in a signal generated by the Arduino microcontroller board [28]. The Arduino Uno board uses a programming function called analogWrite(x), where x’s value can range from 0 to 255. The board automatically sets up the PWM pulses according to that function and sends that pulsing signal to one of the board’s PWM-enabled output pins. We used a versatile and easy-to-use fuzzy logic library for the Arduino Uno board called FuzzyLibrary [29].

Figure 3 shows the resulting fuzzy sets with linguistic variables called “slow”, “medium” and “fast”, as applied by [24]. The variables helped to define an output value from 0 to 255 for controlling the fan using PWM technique. We conducted a software engineering testing method called “functional testing” [31] with our fuzzy logic program to see how it behaves under controlled inputs. In an initial functional test, we used the input value 60 cm for the distance between the computer and the user. As seen in Fig. 3, an example of a crisp output value was 195 (PWM value for the fan), represented with a vertical black line in Fig. 3. This value was sent to the microcontroller board to control the fan speed, which falls well within the “fast” subset with a membership degree of more than 0.5, and it somewhat belongs to the “medium” subset with a membership degree of less than 0.05.

Fig. 3.
figure 3

An example of a crisp output value (195) computed in our fuzzy logic system with an input of 60 cms.

Full size image

We ran another functional test with the input value of 50 cm. This was half of the universe of discourse of the input values defined in our fuzzy logic system ([20..80]). The result is shown in Fig. 4. As expected, the fuzzy logic system rightly defined the output crisp value as 128 (represented as a vertical black line), which fully belonged to the “medium” subset with a degree of membership of 1.0. Interestingly, the fuzzy logic system also determined that the output barely belonged to the “fast” subset, although this did not affect the output value. We believe this happened because of inaccuracies of floating point operations calculated by the fuzzy logic library, but we will need to confirm that in further analyses.

Fig. 4.
figure 4

Resulting fuzzy sets with the functional test.

Full size image

Figure 4 shows that the crisp output value of 128 (shown at the center of the “medium” fuzzy set with the vertical bar) ran the fan connected to the Arduino Uno at medium speed, since 128 is a PWM value. This functional test confirmed the usefulness of the fuzzy logic for controlling the fan in our olfactory system.

4 System Overview

We developed a simple math serious game intended for young learners using the Panda3D game engine [32], to test it with our olfactory interface as a proof of concept. Panda3D has been used to develop serious games, such as in [33]. In our serious game, the player must count some pineapples and type in the resulting sum. If the number of pineapples is correct, the game sends a signal to the Arduino microcontroller board through the USB cable. The Arduino board activates the air freshener and changes the speed of the fan using a fuzzy logic analysis of the ultrasonic sensor data. If the player was seated “Close” to the computer, the olfactory interface spun the fan slower and thus sent a small amount of odor to the player. Conversely, if the player was seated “Far” from the computer, the olfactory interface increases the speed of the fan ensuring the odor reaches the player. The olfactory interface was placed very close to the computer screen. The serious game and the olfactory interface are shown in Fig. 5.

Fig. 5.
figure 5

A participant testing the serious game and the olfactory interface.

Full size image

A closeup of the electronic interface is shown in Fig. 6.

Fig. 6.
figure 6

The olfactory interface.

Full size image

The olfactory interface consists of the following components:

  • An Arduino Uno microcontroller board and a USB cable connected to the computer that ran the serious game.

  • A solderless breadboard for connecting the electronic components to it

  • A 2N2222 transistor, used for controlling the fan rotation speed

  • A 1 kΩ resistor for the 2N2222 transistor

  • A second 2N2222 transistor, used for controlling the air freshener

  • A 1 kΩ resistor for the 2N2222 transistor

  • A 0.1 microfarad capacitor connected to the transistor, to reduce electrical noise.

  • An HC-SR04 ultrasonic sensor, used for measuring the distance between the learner and the olfactory interface and the computer screen.

  • A 12-V PC fan, used for diffusing the odor generated by the air freshener.

  • A 12-V power source for the fan

  • A a Glade Sense & Spray (TM) air freshener, containing a Hawaiian Breeze (TM) spray cartridge, with a smell of pineapple. We disabled the air freshener’s motion sensor and its manual push-button and we connected it to the olfactory interface. The Arduino Uno activated the air freshener by grounding its push-button.

As seen in Fig. 6, we used a small box to raise the fan to diffuse the odor coming from the air freshener more efficiently.

5 Preliminary Test Case

One male player aged 10 years old initially tested our olfactory interface, shown in Fig. 5. We used the think-aloud protocol method [34] to obtain qualitative data on the usability of the olfactory system and player’s verbal comments on the test of the serious game and the olfactory interface. The test showed that the pineapple smell generated by the air freshener worked as an efficient olfactory stimulation in the game, supporting a positive player experience. We found that the Arduino Uno microcontroller board can run the fuzzy logic without affecting its overall performance. The fuzzy logic program occupied approximately 25% of the Arduino’s storage memory, allowing for the control of other sensors and fans that may be added in the future. The fuzzy logic system very rapidly activated the fan and the air freshener in the player’s presence with no reported delays, changing the amount of smell by trying different distances between the player and the computer.

We found that a distance of 50 to 60 cm between the computer screen and the player (which is also recommended by ergonomists for a comfortable computer screen view [35]) was acceptable for perceiving enough odor from our olfactory interface. Further usability testing is needed with additional participants in an educational context of use such as a school computer lab to confirm our findings and our fuzzy sets.

6 Summary

The real world is multisensory and our experiences in this world are constructed by the stimulation of all our senses including visual, auditory, touch, and taste. Despite the importance of all the senses in providing us with and understanding of our world, virtual environments including virtual simulations and serious games, and the human-computer interface more generally, has predominantly focused on the visual and auditory sense solely. Simulating the other senses and the sense of touch and smell in particular can be beneficial (e.g., as previously described, olfaction within a virtual learning environment can increase learning). However, simulating these other senses within the human-computer interface including virtual environments is not trivial and although recently progress has been made with the simulation of touch with the availability of consumer-level haptic devices, less work and progress has been made with respect to simulating the sense of smell (olfaction). In this paper we presented a simple and cost-effective interface constructed using an Arduino Uno microcontroller board and a department store air freshener to deliver a scent to an user of a serious game. A fuzzy logic system was used to regulate the amount of scent delivered to the user based on their distance to the display. As a proof-of-concept, we developed a serious game intended to teach basic math (counting) skills to children. Learners (players) must collect pineapples from the scene and then enter the amount of pineapples collected. As the pineapples are collected, a pineapple scent is emitted from the interface thus serving to supplement or complement the learner’s senses and stimulate their affection and cognition [38]. As part of our proof-of-concept, a 10 year old learner played the game and provided us with feedback regarding the interface and illustrated the potential of the system.

This work is ongoing and we are currently in the process of improving the interface and will then conduct more extensive user testing which will begin with one or more rounds of usability testing following an iterative development process. Feedback and suggestions for improvement from the usability testing will be integrated into the system after each round of testing. Upon completion of the usability testing phase, effectiveness testing will follow where we will examine the impact of the interface on learning (knowledge transfer and retention).

Future work will also build upon our own prior work on multimodal interactions that has examined the influence of sound on visual and haptic fidelity perception and task performance [36, 37]. Fidelity, that is, how realistic the virtual environment that the serious game is centered on must be in order to ensure effective learning, is influenced by multimodal interactions. We hypothesize that increased fidelity may help learning. To test our hypothesis we will evaluate our system in an experimental situation where we assess differences in learning with and without the system. This has potentially significant implications for designers and developers of virtual environments, given that with our current technology, we cannot faithfully recreate olfaction (we are actually far from being able to do so). However, just like sound can influence our perception of visual fidelity, future work will examine the influence of sound on olfaction perception with the goal of examining whether sound can be used to influence our perception of smell and more specifically, whether sound can increase our perception of olfaction perception. This may have large implications for developers of displays as it may allow low fidelity displays coupled with appropriate auditory cues, to be used in place of cost prohibitive and potentially complex displays.