Keywords

1 Introduction

Currently, we are facing an increase in the use of mobile devices while watching television content [1]. The user is exposed to an endless number of stimuli that can lead to an information overload. Thus, to get users’ attention it is necessary to offer them services that meet their needs, with personalized experiences. Considering this trend, the Smartly project aims to expand the range of services offered to the consumer, trying to capture their attention through complementary information between the television and the smartphone. Thus, Smartly aims to fill this gap by encouraging consumers to watch television ads providing them with related promotional coupons. To achieve this, a set of tools capable of triggering notifications, which are associated with the TV content, were developed. When a specific notification appears on the television screen, the user is invited to press the yellow button of the remote triggering an invitation to use a mobile app. Once installed, the mobile application allows users to receive, manage and use promotional coupons related with watched TV adds.

2 Theoretical Context

This paper reports on the Smartly research project that incorporates an ecosystem in which an interactive television (iTV) application and a mobile application are part. Therefore, the concepts of iTV and smartphone are fundamental in the structure and design of the conceptual framework of this work.

iTV combines enriched TV content with the existence of a return channel [2] allowing applications to have interactivity, enhancing their ability to meet user needs. The second-screen concept enriches the TV viewing experience, because users can benefit from mobile applications in parallel with the usage of TV screen [3]. According to a Nielsen’s survey [4] about TV viewing and digital device usage, 35% of respondents look up or shop for products and services after they were advertised on TV. Notifications appear as a connection link of this technological ecosystem: iTV and smartphone. They serve as a link [5], allowing users to be warned that they have new information available [6] and can be transmitted in several forms (visual, audio or haptic) [7]. Although notifications are considered beneficial to user, they are also considered a great source of distraction [8]. However, this is not a reason to be deactivated, users give them great value because without cognitive effort, it is easy to obtain relevant information [9].

According to [10], about 52% of smartphone owners (this number continues to increase) want to be connected to their personal device as well as to television, creating a multi-screen scenario. In that sense, it was necessary to find and describe examples of applications allowing this kind of interactivity, but also applications that notify users in several devices simultaneously.

Following, a set of examples of applications is listed and a brief description and the explanation of main features and functionalities, strengths and weaknesses are presented.

LG webOS TVNotify - The purpose of this application is to enable users to view notifications (for example, from missed calls, messages, emails, or be related to the user’s social networks) on television that commonly are only be displayed on smartphones, So, users do not need to always be close to the smartphone. It offers privacy control settings allowing certain personal information not to be transmitted on television and give the user the power to choose which kind of notification he prefers to see. More recently, the application has allowed to be added to a black list from which we do not want to receive notifications on television [11].

Notifications for Fire TV - Like the previously mentioned application, this allows the user to forward notifications received on the mobile phone to the television. Other features have been added like the ability to make phone calls, send images from other applications, take screenshots, and quickly search for other applications and the ability to customize the notification. The great advantage of this application is to allow privacy mode for specific applications [12].

2ndVision - This application automatically syncs a mobile application with the TV content (using audio fingerprint), showing on the tablet complementary content related to what is being shown on the TV (name of the actors, a brand of a car, information on a location seen on the TV show, among others). The user can find all the information previously mentioned in the feed, filter the contents by a certain category (e.g.: location, weather, etc.) and store them for later use. There is an agenda that allows users to specify, according to their preferences, the programs they want to see later. The application allows also to create notifications, letting users to receive content alerts. The user may also classify the additional information through the rating and share it using e-mail or social networks. All extra content is gathered and presented to the user due to audio recognition features, namely audio fingerprint [13].

Sony Notify BRAVIA is an Android application that, like the applications mentioned above, allows users to choose which of their personal applications can forward the alert to the TV set, as well as to change the privacy settings (between basic and advanced). If they are on the same wi-fi network as the television, a pairing between the smartphone and the Sony TV can be done and notifications can be forward from the smartphone to the TV set. As differentiating factors, this application allows users to view the history of the notifications received by TV and all people on the same Wi-Fi network can connect to it, creating multiple connections [14].

3 Smartly Project

The Smartly project is a partnership between Aveiro University and Altice Labs, aiming to create a set of tools capable of promoting television ads consumption, using mobile devices. To promote this, it offers a feature that allows the user to receive a notification, while watching TV, to install a mobile application. After installing and signing into the application, whenever a user has the TV tuned on the channel that shows an enriched ad, the notification is displayed int the TV set and the user will receive a notification on the mobile application with an offer related to the television content (the ad).

To create this ecosystem, it was necessary to develop a network between a set of components, such as: Content Recognition Engine, MEO Infrastructure (MEO is the IPTV service from Altice Labs), Smartly application, Smartly Database and Smartly TV application. These components and their connections are represented through the Smartly system architecture diagram (Fig. 1).

3.1 General Architecture

The system architecture comprises multiple parts. The “backend” part uses a couple of API’s to allow to manage notifications in the Smartly database and in the Smartly mobile application. This “backend” part is also responsible for checking the necessary requirements for receiving a notification (through Engine Content Recognition and MEO Infrastructure) and to display it in the television (through Smartly TV application). In the “frontend” part a web interface for the backend management (Smartly Web interface) and a mobile application (Smartly mobile application) where user can see and manage his coupons were developed.

Fig. 1.
figure 1

Smartly system architecture

Full size image

The whole notification process comprises different steps that are shown in Fig. 1 and described here.

  1. 1.

    The Content Recognition Engine constantly analyses if the transmitted content has associated coupons. When the system detects that such a content is being emitted, for which there are associated coupons, a notification is sent to the Web interface API, and the next steps are triggered.

  2. 2.

    In the MEO Infrastructure it is checked if the boxes are active and, if so, if they are tuned in the channel where the content is being broadcasted.

  3. 3.

    If the set-top box is active and is tuned to the channel where the content is being broadcasted, this information is communicated back to the API Web Interface that inserts, into the database, information about the notification, the Set-top Box ID and the channel.

  4. 4.

    A notification is sent to the smartphone allowing users to open the coupon in Smartly Mobile App.

The Smartly TV App will make regular requests to the database to check if there are new notifications to show (a). After detecting that exists a new notification, a message appears on the television screen indicating the existence of a new offer and informs the user that is necessary to press the yellow button on the remote to perform the provisioning (b). Provisioning is the procedure that associates mobile phones to a specific set-top box (STB). This procedure is only necessary in the first use or when the user wants to associate a smartphone with a STB (it is possible to have one smartphone associated to n STB). After completing the provisioning, the coupons will be automatically sent to all associated smartphones. If the yellow button is pressed (on the remote of the STB), the provisioning screen appears with a QR Code, which redirects the user (through a URL with the Box ID) to an interface requiring: i) The name of the user; ii) The mobile number to identify uniquely the user; iii) Download the application throughout Google Play store.

In the second step, when the user enters his mobile number and clicks “Next”, the database is updated, to associate the mobile number with the previously registered Box ID. Following the installation, the user is presented with a welcome screen and is required to authorize the necessary permissions for its full operation, namely the access to the mobile number. This permission will register the users to receive notifications, because is necessary to check in the database which Box ID is associated with that specific mobile device. Smartly Mobile App allows the user to not only view their coupons, but also manage the notifications. The Web Interface API, the Mobile App and the TV App interact with the Smartly database, either to save user information, boxes and mobile numbers, as well as user management (mobile number, Box ID, notifications, categories, etc.), notifications and relationships between users and notifications. It should be noted that notifications sent to users’ smartphones are always made according to their customized preferences (they can select the categories of notifications they want to receive throughout mobile application).

3.2 Mobile Application

The Smartly mobile application allows users to manage their coupons and preferences. Figure 2 and Fig. 3 depicts the main screens of the application, where four menu sections are visible: “Home”, “Coupons”, “Help”, “Definitions”.

Fig. 2.
figure 2

Smartly mobile app

Full size image

The “Home” (Início) screen is the application landing page and shows the new coupons, the expired ones and the number of favorite coupons the user has.

When the user chooses the “Coupons” (Cupões) menu item he has access to the “Categories” (Categorias) page, where the coupons are listed by categories where the user has the possibility to search for a specific coupon using the search box on the top of the screen. He can also add or remove a coupon from the favorites or simply see more details about it.

In the “Help” (Ajuda) menu item the user will find the application tutorial. In the “Definitions” (Definições) menu item the user can choose from which categories the user wants to receive notifications, manage the Set-top boxes the user is linked to, turn on/off notifications and see the privacy policy.

Fig. 3.
figure 3

Smartly mobile app

Full size image

4 Evaluation Methodology

Aiming to provide users a rich experience in this ecosystem, it is mandatory to test solutions with the target audience. This can be achieved through tests that consider user experience (UX) and interface design. Several UX definitions are available but [15] they as being associated with the experience and interactivity between the user and a technological device, in other words, are the perceptions and responses that emerge from the interaction between an individual and a technological device during product, system or service usage. It becomes difficult to define this concept, since there are several perspectives on it. However, this is a consequence of three variables: the user’s way of thinking and acting; how the system was developed; and the context in which it was experienced [16]. As for interaction design, according to [17], it covers four main practices: identifying needs and establishing requirements; develop alternative design to meet certain requirements; build interactive versions of the design developed so they can be communicated and benchmarked; and to evaluate what was developed during the process.

The user experience (UX) is still a difficult concept to define because it is associated with diffuse and dynamic concepts: emotional, affective and aesthetic variables. The UX analysis is also very versatile because it can focus on an individual interaction between the user and the application or on multiple interactions of countless users with one or several services [18]. In addition to the definition of UX it is also important to identify the evaluation methods of this concept. [19] emphasizes that if the evaluated product has already been available to users a few months ago allows a better evaluation of the product by them since they have already had the opportunity to try it. On the other hand, when evaluating a prototype, especially on paper, it becomes more difficult to provide the actual context in which the product is inserted to the user. However, sooner UX is evaluated in a product or service, more successful it will be since errors and constraints arising from user feedback can be bypassed in earlier stages, which will be noticeable to users later in use of the product in the daily basis [20].

The methodology used in the tests is divided into four phases (Fig. 4).

Fig. 4.
figure 4

Evaluation methodology

Full size image

In the first phase, the participant characterization was made using a questionnaire based survey, allowing researchers to organize and assemble data faster and more rigorously. In this survey, the user was questioned through multiple-choice questions (with single and multiple answers) and open-ended questions about the personal information (gender, age, literacy, etc.), digital literacy as well as television and smartphone usage habits (including the use of digital discount coupons).

The second phase included a short introduction about the Smartly project and then the user is invited to watch TV while interacting with the mobile application. Researchers asked participants to perform thirteen specific tasks in the Smartly mobile application while watching a TV video that simulates the TV broadcast (the video includes news programs and commercial breaks where a notification informing that a new coupon is available appears).

After finishing the interaction, the third phase of the methodology was started, in which the user must fill out a new questionnaire survey to evaluate the UX/UI.

Finally, the fourth phase of this methodology included an interview to obtain information about user overall impressions.

In the case of Smartly, SUS (System Usability Scale), Attrakdiff and Self-Assessment Manikin (SAM) were used to evaluate the user experience. SUS questionnaire was used to measure instrumental qualities such as effectiveness, efficiency, satisfaction [21]. It consists of a ten-item questionnaire with five response options (likert scale) for participants, from strongly agree to strongly disagree. Attrakdiff questionnaire was mainly used to determine non-instrumental qualities such as aesthetics and visual identity of the application [22]. This instrument is based on semantic differentials and consists of 28 seven-step items whose poles are opposite adjectives (e.g. “good - bad”). Each set of adjective items is ordered into a scale of intensity. SAM scale helped to analyze the user’s emotional reactions, such as pleasure, arousal, and dominance over the application [23]. The SAM scale is based in three diagrams in which the first measures the level of satisfaction between happy and sad, the second measures the motivation from excited/enthusiastic to calm/boring, and the third measures submissive to powerful control. The fact that the range of possibilities varies between opposite feelings allows users to state the level of intensity over each emotion as well as to see whether it has been positive, negative or neutral.

In order to put into practice this methodology and the mentioned scales, the tests occurred in a controlled environment – a laboratory. This gives the researchers the opportunity to be with the participants in an isolated and uninterrupted moment, thus controlling the influence of the context. Frequently, in these moments, a direct observation is carried out allowing the researchers to pay attention to the individual’s interaction with the system. To achieve this there are several techniques: think aloud, cooperative evaluation, protocol analysis and post-talk walkthroughs [19]. In this case, the analysis protocol is characterized by various forms of recording the users’ actions: i) pencil and paper; ii) audio recording to follow the think aloud; iii) video recording to perceive the users’ behavior with the application; and iv) recording of data/problems/notes of what is going on throughout the test. Users were asked to comment their experience with the system after the interaction but with the opportunity to resume the use of the application while they were being questioned by the researchers about the application. This method, called Post-talk walkthroughs, is used when the tasks requested by the researchers are demanding and do not allow verbalization at the moment of interaction by the user, like in the case of this work [19] registration.

5 Results Analysis

After completing the tests with the users, data from both questionnaire and interviews were analyzed.

Most of the subjects who participated in the study were between 18 to 29 years old (13), three (3) participants were between 30 to 39 and five (5) were between 40 to 49.

5.1 Questionnaires

Characterization

According to the participants characterization questionnaire, most of the subjects do not watch TV for more than two hours a day and, when they do, it is at night. When asked what action they took when advertising is announced, the responses were not consistent, because height (8) participants answered “I change to another channel,” six (6) said “I don’t change channels, but I don’t pay attention to it” and six referred “Depends on advertising”, and no participant answered “I see because it might be interesting”. These answers were somewhat disappointing, however, when participants were asked whether they would make an effort to see advertising if the discounts worth it or if they liked it, participants responded positively, saying that this could be the reason to change their attitude. Regarding the categories most significant for consumers, “technology” was the one that got the most interest from the participants with fourteen (14) selections. Additionally, they also found other relevant categories: “supermarket” (5), “fashion” (6), “health and wellbeing” (5), “home and decoration” (7), and “leisure/travel” (9).

When participants were asked if they use other technological gadgets while watching TV and which are they, fourteen (14) subjects mentioned smartphone, four (4) said tablet, two (2) referred computer, and four (4) participants did not use any device. When participants use the smartphone, they handle it during advertising (8), to search for additional information (9), to entertain themselves (11), and to talk with others (2).

In contrast, participants do not give clear feedback about receiving discount notifications on the smartphone, since nine (9) subjects answered that they would not like to gain discount coupons and eleven (11) said they would be receptive to get coupons. The participants who responded affirmatively were also asked to justify their answer and some reasons were: “The smartphone is the device I pay most attention to and therefore it is the most convenient spot to see and use the notifications” and “It is the easier way to immediately access to new discounts”.

When questioned if they would be willing to receive notifications about advertising on their smartphone, half of the users answers positively and the other half negatively. The participants who gave a positive answer, indicated justifications such as: “Because sometimes I may not be with 100% attention on the TV screen”, “It would serve as a reminder”, and “To facilitate information retrieval”.

The use of discount coupons by the participants varies widely. Most of the subjects (11) reports that they use them rarely, and three (3) users use coupons several times a month. The options “once a month”, “several times a week”, and “once a week” only had two selections each. Most of the participants (18) had already used digital coupons rather than paper coupons (2), according to users, because “It does not take up space and it is a more environmentally friendly way”, “Digital coupons are easier to store and organize”, and “Sustainability reasons”. The same reasons mentioned above should have led participants to opt for coupons in QR Code format, but only seven (7) subjects select this option, and vouchers got the same amount of responses. Barcode and alphanumeric coupons had three selections each. These types of coupons are the most popular formats in stores and supermarket chains, the resistance to change in adopting digital formats can explain this fact.

UX/UI Tests

Regarding the evaluation of user experience, the research team used the SAM, SUS, and Attrakdiff questionnaires. The results are listed in Table 1 and discussed in the next paragraphs.

Table 1. Application score according to the SUS, SAM, AttrakDiff instruments
Full size table

The SAM questionnaire allowed to measure the user emotional responses when interact with the application. Three factors were measured with the 5-point rating scale version: satisfaction (1 - low satisfaction, 5 - high satisfaction), arousal (1 - low motivation, 5 - high motivation) and dominance (1 - low control, 5 - high control) [23]. The results in the test were: satisfaction = 4.23; arousal = 3.64; and dominance = 4.63. Considering this, it was noticeable that the feedback and the results of the SAM depict a general satisfaction of the users, although the motivation could be greater (that is why several extra functionalities were proposed). Regarding control, the obtained value was positive and the users feedback showed that in most features the user felt confident and with control of the action, even though there are some improvements to be made.

The SUS questionnaire, as referred in the methodology, is used to evaluate the overall usability of the system. Through the answers given, on a scale from 1 to 5 [21], the researchers perform their analysis by subtracting one from the answers to the odd questions and, in pair questions subtracting five the user response. After that, the values are then the result multiplied by 2,5. After that, the researchers obtained the amount 82.6, which is included in the fourth and last quartile of this scale rating (Fig. 5).

Fig. 5.
figure 5

Representation of the SUS results of Smartly app

Full size image

Attrakdiff allows getting three types of results: the resulting portfolio, the mean value diagram, and the description of word pairs [22]. The result portfolio measures, along the vertical axis of the visualization, the hedonic quality (HQ), while the horizontal axis displays the pragmatic quality (PQ). According to the image generated by the evaluation of Smartly application (Fig. 6), this product has a pragmatic quality of 1,34 (with 0,23 of confidence) and hedonic quality of 0,83 (with 0,25 of confidence).

Fig. 6.
figure 6

Result diagram of Smartly App - portfolio of results

Full size image

The diagram of average values measures the pragmatic quality (PQ), hedonic quality (HQ), both identity (HQ-I) and stimulation (HQ-S) as well as attractiveness (ATT) (Fig. 7). Thus, it shows that the four values are above 0 (on a scale of −3 to 3), where stimulation is the lowest (HQ-S: 0,51) and attractiveness is the highest (ATT: 1,81).

Fig. 7.
figure 7

Result diagrams of Smartly App - average values

Full size image

Finally, considering the word pairs, it is important need to pay attention to results as these can indicate which features of the application are critical and which are well resolved. In this regard, on the positive level, the app was considered very practical, manageable, well structured, complete, and well presented. On the other hand, on the negative level, participants found it common and complicated.

5.2 Interviews

After the questionnaires, the research team interviewed the participants. Some important insights were gathered in these moments. The most relevant are listed below subdivided in two categories: common feedback and unusual feedback.

Some of the user’s common feedback were related to: i) the interaction with the coupons (“The auto play of the carousels does not allow full control and the transitions are too slow.”, “The information on each coupon has too many clickable steps. This becomes annoying and, in addition, the information becomes redundant.”); ii) the way coupons search is done (“Why is there only the possibility of searching in the “all” coupons tab? I would like to be able to search in the entire menu for coupons.”, “It would make more sense that within each category the coupons were listed as in the “all” coupon tab. It would be more consistent, and it would be easier and quicker to find a specific coupon.”); iii) the textual and visual feedback provided by the application (“I think the coupon validation date is missing at the home menu. For me that is an essential piece of information.”, “I realized which was the bookmarks button and how to use it, and I was not surprised by the plus or minus symbol, despite being accustomed with the state color differentiation.”).

Alongside, the unusual ones were: i) “the coupon expiration date should always visible”; ii) “the application should allow the user to choose, in each category, how he wants to view coupons - list or carousel”. Although they can be obvious aspects, they were not implemented at the development stage by the researchers in order to simplify the user interface and to maintain application consistency.

6 Conclusions

With the increasing adoption of second-screen devices as complementary applications while users are watching TV, the importance of finding solutions that can balance user attention between two or more screens becomes even more relevant. The Smartly application proposes a distribution scheme of discount coupons that follows this issue.

The adopted methodology to evaluate the Smartly user experience was divided into 4 phases: user characterization questionnaire, interaction with the application, questionnaires with UX/IU scales and interviews. This strategy allowed to get relevant data for the next phases of the application development and to verify that the application was considered by participants intuitive and easy to handle. These important inputs can help to enhance the advertising efficiency.