Tangible Tourism with the Internet of Things

Abstract

The Internet of Things (IoT) enables new ways for exploiting the synergy between the physical and the digital world and therefore promises a more direct and active interaction between tourists and local products and places. In this article we show how, by distributing sensors/actuators in the environment or attaching them to objects, one can sense, trace and respond to users’ actions onsite. Our research method analysis specific scenarios (case studies) of tangible interaction. We first discuss important issues, which were identified in these scenarios, and are related to log analysis, system usability, and extended models for learning user preferences. Then, the lessons learned in these specific cases have informed the constructive design of a wider scope infrastructure, which is here described and motivated. We envisage the tight integration of localized IoT solutions into a comprehensive mobile information system for tourism.

1 Introduction

Fast-paced advances in the field of the Internet of Things (IoT) make the exploitation of responsive networks of sensors and actuators to enable new ubiquitous information services possible. Nowadays, end-users of tourism information services can take advantage of different modes for acquiring information and satisfying their visiting goals: both through online services (e.g., mobile information guides) as well as through the interaction with physical objects (e.g., smart cards that activate services) and digitally augmented places (e.g. through public displays activated by presence). However, in the tourism domain all these different forms of interaction have hardly been integrated in a holistic system that offers a seamless and personalized user experience across several touch points with tourism products and services. For this to happen, there is the need of a system infrastructure that entails the reasoning over users’ requests and actions monitored by heterogeneous applications, sensors and devices. In such an infrastructure, tourists should be able to seamlessly search for information about points of interest and local purchase opportunities through their personal device. Moreover, they should be advised about the presence of nearby interesting places (recommendations) and related special offers on their smartphones or public displays. Finally, in order to inform their decisions, they should be offered the possibility to manipulate products to purchase while getting detailed descriptions about their unique features.

However, the complexity of dealing with a large application scenario that integrates IoT and ubiquitous information services of several types calls for a multi-step approach. In this paper, we follow a bottom up approach. We first report on a few replicable application scenarios at progressive levels of complexity, which we have implemented. We then illustrate the lessons learned in their analyses and the future work directions that we are now following in order to scale to a full IoT enhanced destination information system. Section 3 of the paper describes the initial simplified standalone IoT installations that were evaluated with end-users to develop and test a light-weight IoT platform that is robust, and easy to update and monitor from remote. Results from the pilot studies are reported in Sect. 4. We then turn to consider how smart things can be integrated into a wide area information infrastructure involving a network of IoT onsite installations interconnected with mobile personalized services accessible from personal devices and with web-based services accessible from personal computers or info points (Sect. 5). This infrastructure includes a centralized, cloud-based management of users’ activity logs and recommendation services which support optimal information selection. The lessons learned, which are summarized in Sect. 6, suggest that a proper design of the interaction with the smart things is fundamental in encouraging the correct usage of the system. In fact, we observed that not all the recorded interactions with IoT installations correspond to a meaningful user behavior, and therefore caution must be exercised when automatically analyzing the collected data. When this is done, learning preferences from low-level behavior data is possible, as our initial study shows, and developing recommendation services for suggesting what to do next is a feasible task. Some reflections are collected at the end of the paper on issues related to scaling-up to a comprehensive scenario of interconnected physical and digital information services.

2 Background

Mobile information services have rapidly become indispensable helpers for tourists with a basic familiarity with technology (Grün et al., 2008; Rasinger et al., 2007; Ricci, 2011). But new forms of data exchange are revolutionizing the way information is distributed. A prime example is the new paradigm of the IoT, which refers to the networks of interconnected devices and objects embedded with electronics that exchange data and cooperate towards a common goal (Atzori et al., 2010). IoT enables new and effective ways for getting situation-aware information and a more direct dialogue between tourists and local products and places, thus providing innovative instantiations of the Smart Tourism concept (Buhalis & Amaranggana, 2014; Gretzel et al., 2015). By distributing sensors/actuators in the environment or attaching them to objects, it is now possible to sense, trace and respond to users’ actions onsite or to cope with the evolving state of the environment (Kubitza et al., 2016; Neuhofer et al., 2015; Guo et al., 2014). Therefore, we are not only able to monitor the information tourists search for, their bookings and preferred topics, but also the places they stop by, the products they get engaged with, and which actions they perform (observe, evaluate, compare, purchase, etc.). This offers the possibility to use a wide range of means for involving tourists in more captivating physical interactions, for personalising the information services more effectively and for better understanding the market needs and preferences.

IoT technology supports, for example, the augmentation of relevant points of interest distributed in the environment (outdoors or indoors) with beacons, i.e., small devices broadcasting low-energy Bluetooth messages encoded with standard transmission protocols (e.g. Eddystone¹). These messages can be sensed by the Bluetooth receiver available in most of the personal devices (tablets and smartphones) and, with the aid of background processes running on the devices, can fire the generation of location-based notifications or feed information to a user model in order to support further personalization of the system generated information (Ng et al., 2017). The possibility to augment physical objects with sensors, detecting when they are moved and manipulated, enables scenarios where descriptive information about objects is presented to users at the very exact time they are inspecting them, hence, stimulating enjoyment and sharing (Shaer & Hornecker, 2010). We have, for example, public displays showing a description when a product is taken out of its display position and is manipulated, with object movement detected via beacons. Or, as it will be illustrated below, one can build an interactive plinth (smart showcase) that provides information about objects that are put on top of it—with NFC tags attached to the objects and recognized by a reader integrated in the plinth. The information offered to the user, while interacting with the objects, can be adapted by explicitly selecting the output language and the type of information (theme) through buttons or other physical gestures. This type of interactive experience has been evaluated successfully in the cultural heritage domain (Marshall et al., 2016), where tangible interaction with exhibit objects, i.e., the encounter with the materiality dimension of museum artefacts or replicas, has the potential to increase visitors’ engagement (Dudley, 2010; Petrelli et al., 2013). But the full application of tangible interaction to the development of novel tourism information services is still in its infancy. In fact, these types of scenarios, as we have mentioned in the introduction, have not been systematically investigated in a wider tourism perspective yet, e.g. when applied to the promotion of local products and in synergy with the other online information of tourism services.

To fill this research gap, the technological infrastructure presented in this paper has been purposefully designed and implemented (i) to speed up the deployment in the tourism domain of IoT installations that support the material exploration of artefacts, (ii) to easily interface with a wider network of digital services accessible through mobile devices and info points, and (iii) to put in place the necessary services for logging both the online user experience and the tangible interaction of tourists, thus enabling future studies on consumer behaviour to evaluate the impact of IoT applications in the tourism domain.

3 Research Method

The exploratory research on tangible tourism reported in this paper was articulated into four stages: the development of a light-weight IoT infrastructure; its use to build interactive installations that allow visitors of cultural/commercial events to interact with artefacts and products; the evaluation of pilot installations with actual users to derive possible design shortcomings, guidelines for the elaboration of log data and requisites for integrating the IoT technology into a larger service network; the extension of an existing platform for web-based and mobile tourism services with the IoT component.

3.1 The IoT Infrastructure

We started our investigation from the development of a light IoT infrastructure that allows to couple the tangible experience of objects with the concurrent presentation of digital information. The technological infrastructure was purposefully developed to facilitate the monitoring, maintenance and update of the system logic as well as the prospective integration into a wider network of the other information services. It includes three main components (Fig. 1): a light local processing unit (Hub), a module managing the content play (Viewer), and one or more preconfigured clusters of sensors (ioBox).

Fig. 1.

The IoT infrastructure required to implement an interactive plinth

The Hub is a miniPC coupled with an output device like a HDMI monitor or a projector and it hosts a server engine that manages the execution of the interaction script. The script monitors the signals received from the sensors, decides which content should be presented and controls the output display by sending messages to the Viewer. It is possible to manage the start/restart of the server engine running on the Hub and to control the download/update of the script through an application running on any Android device. This makes it very easy to test the same hardware with different interaction scripts or with different multimedia content even when the IoT installation is already deployed in place. The preparation of the media content (selection of the files and their semantic annotation) is done with a bespoke authoring tool that can be used also by users with minimal technical background (e.g. tourism or cultural heritage professionals) (Risseeuw et al., 2016). The ioBox (which is a replicable hardware setup with sensor capabilities) includes an Arduino MKR1000 with wifi connectivity and supports various sensors: an NFC Reader, up to five touch buttons, a proximity sensor, temperature/humidity measurement and a light detector. Different ioBoxes can be combined together to create complex scenarios (e.g. with multiple NFC-readers). Finally, the MQTT protocol is used for signal exchange between the components. Should the plinth be installed in a setting where good internet connection is available, the logs generated by the IoT installation, and in particular distinct information about the actions performed by the users on the augmented objects, are broadcasted to a central server for real time monitoring of its usage, detection of anomalous functioning and remote maintenance. Otherwise logs are stored locally.

3.2 Smart Installations for Experiencing Objects

Figure 2 shows one of the prototypes that we deployed in the Museo Storico Italiano della Guerra (Rovereto, Italy) since January 2017. Here, visitors have a direct material experience of original historical objects (artefacts from the WWI period) combined with information provision. It comprises two distinct areas, one to showcase the objects and one central active area. An initial presentation message invites visitors to select one object and put it within a clearly marked area. This starts an audio and a graphical animation describing what the object is, how it was made, and what it was used for. The visitor is invited to pick up and touch the object, observe it closely, possibly handing it to visit companions: these actions do not interrupt the presentation. Two buttons are available to select the output language (Italian or English).

Fig. 2.

Museum visitor manipulating exhibited objects while hearing their description (left) and choosing a different language of presentation via pressing a button (right)

Figure 3 shows a second prototype that was developed to test similar IoT technologies to those illustrated above, but in a commercial setting where different challenges come at the forefront: the need to homogenize the installations to the company brand strategy; the requirement to attract and engage visitors to foster subsequent interactions with the salespersons; the utility of collecting product popularity statistics to initiate further marketing campaigns. Two exemplars of this prototype were tested with end-users during the four days of Vinitaly 2017 fair (Verona, 9–12 April 2017). Bottles of wine were available to visitors for a closer inspection of the packaging and for activating multimedia descriptions when bottles were placed onto wooden boxes of the selling winery, which were augmented with an RFID reader and buttons. Touch areas on the box surface were available to select the preferred output language (Italian, German, and English) and the type of information to display (information about the winery, the land of grapes growth, and the properties of the wine). Although very specific, this scenario can be generalized to many similar situations in which: there is a collection (catalogue) of objects with related information; stakeholders (e.g. retailers, exhibitors in fairs or markets, museum curators) are interested in conveying detailed information about the objects (e.g. technical features, organoleptic properties, manufacturing techniques, or their history); end-users are interested/need to learn about the objects; the physical engagement with the objects might improve the user experience; it would be difficult for the organization to provide personally all the details to individual users.

Fig. 3.

IoT installation for product presentation at the Vinitaly 2017 fair (left) with a detail of its internal technical setup (right)

Within the two pilot installations described above, the IoT infrastructure proved to be robust, and easy to update and monitor remotely. The interactive plinth illustrated in Fig. 2 has been running for 8 months now, with only minor technical problems related to the instability of the internal museum wifi network that often prevents the connection from remote. The two interactive plinths illustrated in Fig. 3 were made available to the public for 4 days in a very crowded environment (the fair had about 128,000 visitors in total) with some problems on the usability side (as discussed in Sect. 4 below) but not on the technical side. The infrastructure scalability and modularity were also tested during the recent refurbishment (June 2017) of an exhibition at the Artillery section of the Museo della Guerra where a network of 5 different interactive stations (with multiple NFC readers each) provides multimedia information on a historical theme through object activation.

4 Lesson Learned from the Pilot Studies

When IoT installations support tangible interaction with products/objects (as those illustrated in the previous section), are deployed in shops, street markets, fairs, or exhibitions, they become a valuable source of information on activities of the consumers during the decision-making process. In fact, several automatic measures can be computed over the system logs, i.e., the snippets of data that register users’ actions and events. Similar to web analytics of online information services and data mining of mobile apps usage (Liu, 2007; Pitman et al., 2010), the log of IoT installations can be analyzed for different purposes: (i) at run time, to collect information useful to understand the preferences of individual visitors and dynamically improve the relevance of the provided information and to feed user models that bootstrap the personalization of other online or mobile services used at following stages (Petrelli et al., 2017); (ii) to predict visitors’ next actions and movements for better tuning the system behavior, e.g. when recommending sequences of POIs to visit (Hashemi & Kamps, 2017; Massimo et al., 2017); (iii) at periodic intervals, to compute statistics on system usage and visitors’ preferences that may be of interest to the tourism/cultural organization for tuning marketing strategies or to understand the impact on their visitor experience; (iv) to identify system faults or usability problems.

In the pilot IoT installations described in Sect. 3 the following logs were collected: when an object (NFC tag) is placed in or removed from an active area (NFC reader) of the interactive plinth; when a button associated to a thematic or language choice is pressed; when a media file has started playing; when a media file play comes to an end; and when a media file play is interrupted. These types of logs allow to identify: the objects the visitors spent more time with; the types of information (themes) that were more frequently requested; which information is more frequently interrupted; but also, more advanced models of system usage, such as preferred sequences of actions and topics’ relative importance for the users. For example, the logs collected at the multi-station installation at the Artillery section of Museo della Guerra were successfully used in an experiment addressing the challenging problem of learning the preferences of users (what item is chosen next and why) from low-level behaviour data by focusing on modelling and learning these preferences in a sequential decision-making problem, and using a novel machine learning technique called Inverse Reinforcement Learning (Massimo et al., 2017).

However, the experiences made in the first pilot studies reveal that caution must be exercised when automatically analyzing the logs of IoT installations, as not all the recorded interactions correspond to a meaningful user behavior and that extracting user interests and preferences from raw interaction data is not straightforward. Several issues need to be carefully taken into account: some irrelevant data may simply correspond to user attempts to understand how the system works or what type of information is available; usability problems might cause mistakes or repeated actions; the context where the IoT installation is placed might influence how it is used, e.g. crowding might urge users to free the installation earlier than actually desired. These phenomena are clearly confirmed by the logs collected during the four days of the Vinitaly 2017 fair, which was our first attempt at deploying the IoT infrastructure in a commercial setting with high volumes of people and with a shallow user motivation for using the installations. The log analysis conducted over the 5600 recorded events of users’ actions shows that the vast majority of the logged events (88%) are button pressing (for changing the output language or choosing the type of presented information) and only 335 events (6%) are selections of a new object to be described (6 different bottles of wine were available for the first plinth and 3 for the second). There are several sequences where the same “button” (which is a visually marked active area) is pressed repeatedly within a few seconds, with users apparently just testing how the system works or not sure they had pressed the desired option successfully. The latter usability problem is related to the adoption of the original wooden boxes used to ship wine bottles as the physical casing for the system: it came out that augmenting the wooden surface with touch capabilities was not completely intuitive for users who did not receive explicit feedback on the fact that the touch-active area had been successfully pressed apart from the media play start. The reduced number of product changes on top of the plinth is instead probably due to the location where plinths were placed, that is, very close to the entrance of the two selected fair booths with flows of visitors not favouring the stop and tranquil use of the system: when the second plinth was moved to an inner location, the higher quality of the recorded logs confirms a more proper system usage.

5 Towards an IoT Enhanced Destination Information System

In the final step of our research, we addressed the challenging problem of generating the full design of a Destination Information System that can leverage the new data and services enabled by an IoT infrastructure. Smart installations for experiencing objects have the potential to increase the time spent by visitors in learning about local products and places and to enhance their engagement in the overall tourism experience. To take the most out of this opportunity, however, it is necessary to integrate and harmonize the IoT-based services with the other existing destination information services. The experience gained with the specific IoT installations, described earlier in this paper, has allowed us to define the building blocks of a more general and comprehensive infrastructure. We are currently implementing an information system in which we will validate the exploitation within a unique network of services of: (i) interactive stations in shops for the tangible experience of objects and products, (ii) beacons for optimal visitor localization and identification, (iii) information kiosks, and (iv) mobile services that exploit the logs of all the visitors’ interactions to personalize the information delivered on request or in a push mode. Figure 4 illustrates how these different types of technology are interconnected in a unique network.

Fig. 4.

Scenario integrating interactive showcases, physical info points, beacons and mobile services

Fourteen existing physical info points (kiosks with touch screen) installed at hotels and tourist offices in the tourist area of Val di Fiemme (Italy) and providing information about local events, accommodation, restaurants, points of interest, natural trails, and transport, have already been integrated with the central cloud platform that orchestrates the server-side data services and collects all the interaction logs produced within the network. The same server-side cloud platform supports a mobile app implemented for IOS and Android mobile devices that complements the information provided by the physical info points. The app is a map-based mobile tourist guide featuring information exploration and filtering by category, with additional functionalities for keeping a diary of the travel and receiving personal suggestions for interesting things to do and products to purchase. The logs of the users’ interaction with the mobile app are collected and sent to the central server in order to feed a recommender component that helps filtering, reordering and pushing information which is deemed relevant for each user according to what they have searched so far, the places they have been to and their profile information. We are now completing this infrastructure and deploying four pilot interactive showcases at handicraft shops, which are based on the IoT platform described in Sect. 3.1, to allow visitors inspect and learn about products. About forty beacons will be distributed at important points of interest (monuments, museums, parks and natural trails) and at selected shops selling local specialities. Beacons will also be used to mark the location of physical info points and interactive show cases for the purpose of identifying the current user interacting with them: when the personal device of the visitor detects a nearby beacon associated to a kiosk or to smart show case a log is sent to the central server that associates the subsequent interactions to that user. These log data complement what is separately collected by the mobile app: the sequence of pages the user has navigated to; whether the search filters have been opened to change some of the options or a string of text has been searched for; whether the user has requested to see more products similar to the currently displayed one; when a product or a point of interest has been bookmarked or rated; when a beacon has been detected; whether the user has opened a pushed notification and performed a related action.

The implementation efforts are based on the extension of the existing Suggesto Marketspace platform for electronic tourism, already supporting the flexible creation of web sites for DMOs and their companion mobile apps and info points, with the IoT infrastructure described in Sect. 3 and with the management of beacons (Fig. 5). New recommendation algorithms are being implemented to reason over the extended set of logs generated online and onsite. We have already implemented a core user modeling approach that learns user’s preferences in sequential decision-making scenarios, and can be used to suggest items to be consumed in a sequence. Predicting (or recommending) a full sequence of objects is appealing in the considered scenarios where the user is actually faced with multiple choices, e.g., which Point of Interest to visit next or which objects to manipulate next. User preference learning is made possible by (1) exploiting the mentioned observations of user’s actions while the user is consuming items (POIs or objects), and (2) leveraging item’s descriptive features and environment’s contextual information.

Fig. 5.

The logical components of the Suggesto Marketspace platform for electronic tourism (in light grey the new components being integrated)

In order to reach this goal, we have adopted an approach based on Inverse Reinforcement Learning (IRL) (Ng and Russel, 2000). IRL allows the recommender system to learn a policy that dictates for each possible state of the user interaction with the environment the best action the user should perform (which object to consume). This policy fully describes the sequential preferences of a user or a group of users and can generate relevant recommendations for these users, as well as, new users deemed as similar to the observed ones. For instance, when facing a new user, the system can reuse a model learnt for the users who have a similar information need (goal) or belong to the same socio-demographic group. IRL accomplishes user’s preference learning by discovering the importance weights of features of items at disposal for being consumed (more precisely, features describing the state of the user). These weights define the utility (reward) a user gains by consuming items.

We have tested our general approach for learning the user model in the context of the multi-station installation at the Artillery section of Museo della Guerra (mentioned in Sect. 3.2), where a recommender could suggest which media items to consume next by reasoning on the logs of IoT interactions. In particular, we studied how visitors’ preferences can be learnt, by relying on the observation of the users’ consumption of the available media. The experiment results show that, by just using low-level behaviour data, our approach can learn users’ preferences (reward function) and the policy adopted by users when consuming items sequentially (Massimo et al., 2017).

6 Conclusions and Further Research

The experience made with the pilot case studies of smart showcases and the analyses of the collected log data allowed us to derive some design guidelines for the integration of localized IoT solutions into a comprehensive mobile information system infrastructure for tourism. First of all, the importance of an appropriate design of the interaction and the physical affordances that are offered to users should not be underestimated. Visitors can successfully use smart installations only if: they immediately understand how they work; they perceive a value and pleasure in touching and manipulating physical objects; they have a motivation for learning about objects that stimulates their information exploration both at the physical and the digital level; and smart things are placed in a proper space that facilitates a tranquil usage. A second observation relates to the reliability of the log data automatically collected by the smart things. Noise in the data can be easily introduced, for example, by fast repetitions of the same action (users making unprecise gestures), fast-paced random actions (users exploring the system functioning), very short interaction sessions (staff testing the system at the switching on), but also very long interaction sessions (system failing to recognize session end). Should the IoT installations be integrated in a larger network of information services with a centralized recording of action logs and personalization services adjusting what is shown or pushed to visitors, filtering methods need to be implemented to remove the noise in the IoT logs before this information is used by any user modeling and recommendation algorithm. When this is done, our analyses show that learning preferences from low-level behavior data is possible and that recommendation functions can be reliably implemented to predict and suggest sequences of objects or points of interest that may be appealing for visitors.

One limitation of these findings is related to the small scale of the tested case studies. The scale-up to a comprehensive scenario of interconnected physical and digital information services certainly poses additional challenges related to: (i) the heterogeneity of the data collected at the different touch points of the visitor with the system (IoT installations, mobile app, info points) that are representative of different information goals and contexts of use; (ii) the greater sparsity of the data (both with respect to time and space) that increases the complexity of learning models of users’ preferences and behaviours; (iii) the need for an overarching strategy to design a coherent user experience that seamlessly spans across different services. To further investigate these challenging issues, we are currently completing the implementation of the overall infrastructure discussed in Sect. 5 which involves a network of four pilot IoT installations in shops and forty beacons interconnected with mobile personalized services accessible from personal devices and fourteen info points. The scenario will be tested on the field with the support of the Val di Fiemme DMO in Winter 2017.