Keywords

1 Introduction

Museums have been subjected to important changes in the approach they use to involve visitors, also by using digital technologies. So, they are implementing a digital revolution with the aim to change the way people relate to and visit the museums.

Among the other trends, storytelling and interactive exhibitions are two of the most used approaches used to make the exhibition more and more interesting for the users who visit it. So, from the traditional concept in which museums and exhibitions were “show-and-tell” spaces [1], in this new scenario they are becoming spaces where it is more and more possible interacting with pieces of arts.

There are several approaches used to make museum exhibitions interactive. One of the most used and easy-to-implement approach used to carry out this process of innovation is the digital interaction. Specifically, Virtual Reality (VR) and Augmented Reality (AR) methods can be effectively used in the context of a museum exhibition to support both storytelling and interaction. In particular, by means of AR applications, visitors can enjoy immersive experiences in which the content, the history and the meaning of pieces of art are digitally added onto the real artifacts. Indeed, Augmented Reality is nowadays more and more used for museum exhibition, to add virtual contents to the real ones and “increase” the reality. So, pieces of art are able to self-explain their meaning, the story of their authors, the correlation with other pieces of art and so on. Conversely, in the case of VR methods, these are usually used to re-create historical sites with VR environments to “immerse” the users in ancient historical periods.

The primary objective of the use of these technologies is to make the visit of museums much more engaging, immersive, and suitable for different types of visitors. In particular, the common idea is to “extend” the target audience also to those who are not experts in art (or in the topics described in the exhibition) who, through the new co-communication methods, can better understand concepts often considered as very complex. In this sense, in fact, the possibility of viewing the history of the pieces of art and their meaning instead of reading or listening to the explanation, or rather of looking at images/animations instead of reading texts, reduces the need to have experienced and knowledgeable users, and increases the possibilities of learning and remembering information. In the “non-expert users” category, it is also possible to include children, who are often fascinated by science, nature and art, but hardly interested in static representations that are very difficult to understand.

New exhibitions aspire to raise acknowledge and curiosity among the visitors, especially children and teenagers. In fact, interactive exhibitions are becoming more and more appreciated and effective, involving people through different senses and calling them to action. They need to perform an action in order to see a corresponding reaction inside the environment or in the object shown.

Moreover, nowadays the user experience is one of the fundamental features of museums and exhibitions. Indeed, the user experience is strictly linked to the user interaction with pieces of arts and objects at physical, perceptual and cognitive levels. This means that it includes also the design of multisensory experiences that also satisfy sight, touch, hearing and smell senses in order to improve the exhibition understating and the user’s engagement. In particular, the sense of smell plays an important role, since it recalls memories and deep emotions both at conscious and unconscious level. Thanks to these effects on humans, smells can help the customer to feel involved into and emotionally linked with pieces of arts.

The purpose of the research work presented in this paper is to design and develop interactive multisensory applications (based on sight, hearing, smell and touch senses), which can be used for improving the user experiences of cultural heritage exhibitions. Particular attention has been devoted to the integration of olfactory stimuli in the overall experience and their integration with the other senses. Specifically, the paper presents two case studies in which multisensory AR experiences of cultural exhibitions have been designed and developed. In both cases, the experiences involve the sight, hearing and smell senses and aim at improving the visitors’ engagement in the exhibition and their understating of its content.

2 Literature Review

2.1 Multisensory Perception

Psychological and neuroscientific research has shown that perception is always multisensory: whenever we interact with the external world our brain integrates information from different sensory modalities as a function of certain binding principles [2].

The sense of sight has been thoroughly studied in the years [3]. Concerning the sense of hearing, several research studies have demonstrated that sounds can be used to elicit emotions in users [4]. The sense of touch is the first to develop in humans, well before vision [5]. In a general sense, touch plays an important role at both the emotional and perceptual level, even if the user is not aware of its use [6]. The sense of touch is important in the interaction with other people, but also when we interact with objects.

The sense of smell is one of the main means through which humans investigate the environment [7]. Odors can influence moods [8], emotions [9, 10], and memories [11]. Odors can induce activation or relaxation states in people [12], and can also stimulate cognitive abilities [13, 14]. Concerning the link between odors and memory, many studies have been carried out from different points of view, such as basic science, psychology [15] and marketing [16].

In modern cultures, the sense of vision is hegemonic [17]. So, Western society has been defined as “the society of olfactory silence” [18] and, up to a few years ago, there was the tendency to remove any unpleasant odor. It is only in recent years that the importance of smell on the rise, especially in regard to its potential to stimulate emotions and create positive product experiences combined with other sensory modalities. This property has been addressed especially in the marketing research field, in which many studies have been carried out on the use of odours for influencing customers’ moods in shops, and for communicating information about products [19, 20].

2.2 Olfactory Displays for Virtual and Augmented Reality Environments

In the last years, several studies have focused on the integration of odors in Virtual and Augmented Reality environments by developing devices named Olfactory Displays.

Various technologies are used to develop an olfactory display. These are categorized by scent generation methods and scent delivery methods [7]. Several research works concerning olfactory displays have been presented in literature [21,22,23,24].

The authors have developed some prototypes of Olfactory Display and have evaluated their performances in several applications for products evaluation [25], for improving the drivers’ attention level [26], for heritage studies [27], for improving the reading experience [28], and for improving virtual tours for the tourism sector [29].

Even if olfactory display devices are still evolving and some technical issues still remain unsolved, some companies have developed and sell olfactory displays [30]. Recently, some new devices have been launched on the market. For instance, Vaqso is a device that adds scents to the VR experience [31]. It can be customized by the user, who can feel the scent in conjunction with the scenes and actions on VR. The Olorama scents generator provides user a multisensory experience with wireless scent dispensers on VR/AR and home projection systems [32]. Feelreal simulates several scents to immerse the user into virtual worlds [33] by using a scent generator, which holds a replaceable cartridge containing nine individual aroma capsules.

2.3 Multisensory, Interactive and Digital Exhibitions

Nowadays interactive exhibitions in museums are becoming broadly common, thanks to their flexibility of application and effectiveness in engaging visitors in the experience. So, visitors are becoming active “consumers”, and are able to define their own journey and experience at the museum. One emblematic example are interactive exhibitions for younger audiences, and specifically for children. In this case, museums frequently allow young visitors to “play” with exhibits, in order to make learning new knowledge an interesting and pleasant experience. One example is The Children’s Museum of Manhattan, in New York. In 2016 the museum developed the “America to Zanzibar” exhibition [34], in which children were able to touch and experiment with artifacts of Muslim culture. The main idea was to create a multisensory experience to help children understand the Muslim culture using colors, images, objects and smells.

Concerning the use of digital technology, and specifically VR and AR applications, people are becoming more and more familiar with hybrid realities that make it easier to use this technology in public events and for cultural heritage purposes. Some museums present digital labels or touchscreen tables with texts and information as “side elements” of their exhibitions. Other museums, such as the MONA Museum in Hobart [35], effectively rely on the “O” device and application that replaces wall texts in the museum and represents the only way visitors can interact with pieces of art. Another interesting example is the Cooper Hewitt Pen at the Cooper Hewitt Smithsonian Design Museum [36]. In this case, the “Pen” is used by visitors to ‘collect’ objects around the gallery, which can be explored in further details at the end of the visit (by using interactive tables) and also online. At the Casa Batlló in Barcelona, visits are carried out by using an AR guide, which provides a dynamic and semi-immersive experience [37].

In many other cases, museums develop VR and AR experiences of some special exhibitions. Among the others, in 2010 the AR exhibition named “WeARinMoMA” by Sander Veenhof and Mark Skwarek has been held at the MoMA in New York [38]. The exhibition, performed within the context of another exhibition, allowed the visitors to choose what to see thanks to an application on their smartphones. In 2015, the British Museum’s Samsung Digital Discovery Centre (SDDC) held a Virtual Reality Weekend in the Museum’s Great Court, offering the visitors the possibility to enjoy VR environments via immersive and non-immersive devices [39]. Visitors were able to explore a virtual reality Bronze Age site, where three-dimensional scans of objects were placed. Moreover, interpretations of how these ancient objects might have been used in the past were presented. In 2017, the Natural History Museum in London featured in an interactive VR experience in which visitors were able to virtually handle fossils and other pieces from the Museum’s collection in a one-on-one interactive experience [40].

Finally, concerning the use of stimuli from different senses, this approach can modify how visitors interact with and interpret the exhibition and its content. In this field, several experiences have been developed by using one or more senses at time. In 2012, the sense of touch has been embodied in the light installation of Takahiro Matsuo [41]. His interactive Aquatic Colors installation emulates a glowing, underwater sea filled with jellyfishes, which respond to the movements of visitors. The use of smells for cultural purposes is rare and still experimental, but some exhibitions that integrated olfactory displays have been presented in the last years. In “The Art of the Scent (1889-2012)” exhibition at the Museum of Art and Design in New York scents have been celebrated as a true artistic media [42]. The “Multisensory Met” exhibition at the Met Media Lab Intern Expo aimed at creating a multisensory environment to enhance the overall effect of pieces of art. So, they created more fulfilling experiences for museum visitors, which are satisfactory for multiple senses. Different multisensory products have been used: booklets, sculptures, clay replicas, scratch-and-sniff paintings, scented powder materials and sound paintings [43].

3 Case Studies

In this section two case studies of multisensory AR experiences of cultural exhibitions are presented. In both cases, the experiences involve sight, hearing and smell senses (in one case also touch) and aim at improving the visitors’ engagement in the exhibition and their understating of the content.

3.1 The Stati d’animo - A new AR olfactory experience Case Study

The aim of Stati d’animo – A new AR olfactory experience case study has been to augment the experience of the user of an exhibition related to moods by using an AR application, tactile stimuli and olfactory displays for allowing him/her to be more immersed in the exhibition subject. The AR multisensory system intends to contribute in increasing the visitors’ sense of presence in the environment and awareness of the meaning of each mood.

The case study consists of mixing Augmented Reality technology and simulated smells in the Palazzo dei Diamanti Museum in Ferrara (Italy), where the Stati d’animo exhibition took place in March-June 2018. The topic of the exhibition was a critical research of the origin and growth of moods and feelings in paintings in Italy in the 19th-20th century. The exhibition included many paintings focusing on different moods, one for each of the 12 areas of the museum. What was evident was the presence in every canvas of something related to a smell or to an olfactory sensation (Fig. 1). Specifically, in each room there was a link to a couple of smells that suggested a precise feeling.

Fig. 1.
figure 1

The second room of the exhibition – Involve the observer – gives a big suggestion to the viewer about different smells, in which many of them were used in the past from artists as drugs to augment the perception of reality.

Full size image

So, the case study Stati d’animo – A new AR olfactory experience has the aim to give visitors a new and different way of comprehension of art in museums. Using AR technology and olfactory devices (developed by the authors and described in [29]) to solicit visual and olfactory stimuli creates immediately an exciting memory in the visitors’ mind. Thanks to the specifically-developed AR application and the olfactory experience, when visitors perceive the smells, their feelings and emotions are amplified. In fact, smells let them remember more the different feelings experienced when visiting the exhibition. In this way AR and smells are not used just as new experience means, but also as memories enablers.

Concerning the layout of the exhibition, the rooms tell stories are about the introduction of physiognomy study, i.e. the use of drugs to amplify experiences in life, melancholia and alienation, contemplation and empathy, the musical inspiration, communion and harmony, voluptuousness and feral instincts, fusion and ecstasy, happiness and enthusiasm, atmosphere and affection.

The AR multisensory experience consists of two main components, set up in all rooms.

The first main component consists of 3D printed objects – totems placed in big rooms, and frames placed in smaller rooms – accommodating inside olfactory displays that deliver odours if the visitor brings his/her hand close to an LDR (Light Dependent Resistor). The kind of odour released in the room has been carefully selected to create a link with the mood and augment the perception of the content of the painting. Each odour is a combination of two or three fragrances, according to a balanced creation of “perfume notes”. For example, the fragrance used in the room about Fusion and Ecstasy is a mix of rose and vanilla, a mix of tobacco and brandy for Involve the observer; lemon and cypress for Melancholia and Alienation; sandalwood and lavender for Contemplation and Empathy.

When triggered, the system also plays a music representing the mood of the room. Each 3D printed object used in the various rooms is modular and has a particular and suggestive shape, well related to the mood of the room. For example, in the room about Contemplation and Meditation, the pillar has the shape of an impoverished ionic column, representing the literal meaning of contemplation (from Latin, “cum” and “templum”, delimited celestial space).

The second main component of the system consists of AR applications displayed on tablets or smartphones, which allow animating the most representative painting of each room, displayed on a wall. For example, in the room about Contemplation and Meditation, the most representative canvas of this mood is Ave Maria a trasbordo made by Giovanni Segantini in 1886. So, the authors developed an AR application in which the boat is moving (Fig. 2). A recorded voice accompanies the animation, telling the story of the piece of artwork and explaining the meaning of the room.

Fig. 2.
figure 2

The Ave Maria a trasbordo canvas, which has been animated in the AR application and the A scaled prototype of the totem about contemplation and empathy made with 3D printed material. In the background the Arduino Uno system connected with the spray diffuser in it.

Full size image

The system consists of interactive devices, i.e. columns and frames displaying stories about the paintings on show in the room, and including olfactory displays delivering fragrances. The interactive devices are connected to Arduino Uno and AdaFruit boards, and an LDR sensor triggered when occluded (Fig. 3).

Fig. 3.
figure 3

The architecture of the Stati d’animo - A new AR olfactory experience system

Full size image

The devices have particular shapes according to the mood of the room and are 3D printed with different materials. In order to suggest to the visitor a tactile interaction, some particular areas are printed with a softer material, similar to rubber, and highlighted with a different shade of the colour of the painting.

When the visitor touches the device, the fragrance is delivered in the environment and sounds related to the mood of the room is played back.

About the creation of the AR system, Unity3D and Vuforia software tools have been used. Specifically, the animations of the paintings have been developed in Unity3D, and then the Vuforia software has been used for the development of the AR application. In the AR application, one object per room is used as marker. So, in each room the AR application is able to recognize the frame (or the column), which triggers the animation related to the most representative painting of that room, displayed on a wall.

3.2 The Senses of Religions Case Study

Th Senses of Religions project aimed to develop a multisensory experience to improve knowledge and elicit curiosity about religions with an interactive approach that lead visitors inside the designed environment. Religion is one of the most important side of our history and origins; by analyzing it, it is possible to find the meaning of different behavioral patterns, usages, cultures and actions. So, the Senses of Religions case study has the aim to give visitors a new way of comprehension of religions. The authors decided to use a cultural and didactic approach, with the aim of describing the meaning, the history and the purpose of each religions, avoiding any subjective opinions and judgments. The use of odors in this contest is particularly important, thanks to the broad use of odors in religions (such as the use of incense in Christian religion). So, by using AR technology and olfactory devices (developed by the authors and described in [29]) it is possible to recall memories in the visitors’ mind and create a cultural interest. The storytelling approach has been used to engage the user during the experience, and the possibility to interact with the application has been designed in order to make him/her live a full experience. In this case study, the storytelling concerns the main key elements and the history of religions.

The exhibition starts from a hall where the visitor finds five doors, one for each religion: Christian, Hinduism, Buddhism, Islam and Traditional Chinese. Above each door there is an illustration, representing a religious descriptive image that works as a marker for the AR application. Framing each marker with a tablet or a smartphone recalls the AR application with a representation of the world. For each marker (and door), a different area of the world is colored, which corresponds to the area of the world in which each religion is spread the most. Behind each door, visitors find a room dedicated to each religion. Moving from one room to another one is a very important moment in which the hall plays a key role to reset to a neutral point the nose of the visitor from the previous room.

One of the five represented religions is Buddhism. The storytelling of this religion is based on Siddhartha Gautama’s life, the path he followed before becoming the Buddha.

Moving in the room, the first illustration to start is the Buddha itself, and the last one is the Bodhi tree, which is the place where Siddhartha became enlightened and received the supreme and final wisdom, hence becoming the Buddha (Fig. 4).

Fig. 4.
figure 4

The layout of the storytelling of the Buddhism room

Full size image

Two olfactory displays are connected to the two Image Targets: “Holy man” and “Bodhi tree”. Framing the Holy man marker, the olfactory display spays a smell of sandal, related to the 3D model, involving the spectator deeply. Lastly, when the Bodhi tree marker is framed, the olfactory display sprays a smell of Zen garden, which recalls in the visitor’s mind the ideas of meditation, contemplation and peaceful. The olfactory displays are located inside the marker support (a totem) with a physical inclination of 90°, hidden from the visitors (Fig. 5).

Fig. 5.
figure 5

The Holy man marker (a conch shell) on the totem, which hides the olfactory display, and the AR application running.

Full size image

For what concerns the AR application, the Senses of Religions case study has been developed by using Unity3D and Vuforia software tools. The interaction between visitors and the application has been created according to the storytelling. Specifically, for some objects in the path, visitors are required to interact with the digital device, listen to the story, touch the screen while immersed in the scent emanated from the olfactory display. In the AR application, five markers per room have been specifically developed. So, in the hall and in each room the AR application is able to recognize five markers, which trigger the animations related to the area of the world in which the religion is spread the most (for the hall) and related to the storytelling of the religion (for the other rooms). A recorded voice accompanies the animations, presenting the meaning of each key element. For what concerns the olfactory displays delivering fragrances, these are connected to the AR application by using an Arduino Uno and AdaFruit boards, and a Bluetooth module. Each time the visitor frames an image target linked to the scent, a script is triggered in order to send a signal to the Bluetooth module that activates the corresponding olfactory display (Fig. 6).

Fig. 6.
figure 6

The architecture of the Senses of Religions system

Full size image

4 Conclusions

The research work presented in this paper focuses on the possibility to improve the user experience of cultural heritage exhibitions through multisensory AR applications (based on sight, hearing, smell and touch senses). Specifically, two case studies have been designed and developed with the aim of improving the visitors’ engagement in the exhibition and their understating of the content. Particular attention has been devoted to the integration of olfactory stimuli and their integration with the other senses. Indeed, the olfactory stimuli have been designed with the aim of impacting on the subconscious and conscious levels of visitors to stimulate memories and emotions.

These case studies perfectly match with some of the most important trends used for innovating museum exhibitions, which are digital interaction and storytelling. More, while Augmented Reality is nowadays commonly used for museum exhibitions, the use of the sense of smell and/or multisensory approach is still a rare phenomenon. On the opposite, the integration of scents in these interactive AR applications can be effectively used to create an emotional link between the visitors and the pieces of art, to improve the users’ experience and the users’ memory of the exhibition. So, in an envisaged scenario, multisensory AR applications similar to those presented can be developed and implemented in a wide range of museums and exhibitions, for events related to cultural heritage and can be also remotely experienced by the user at home for virtual visits of museums.