Keywords

1 Introduction

The growing variety of mobile devices, their characteristics and features is enormous, and developers need to be aware of all restrictions inherent in those devices. A common development approach is to reduce the delivered content and strictly adhere to basic standards in order to support as many devices as possible, at the expense of cutting-edge features. Another approach is to customize the design to meet the requirements of those devices that offer the best possibilities (e.g. user base, distribution system) for a business (Wessels et al. 2011).

Especially in the context of tourism, we can witness an incredible development of mobile services supporting tourists in finding information on touristic products (e.g. hiking routes, sightseeing), offering location-based information or tour guides with online check-in capabilities, booking possibilities via mobile apps, weather information, travel blogging, social communication and sharing, and rating and reviewing of touristic products on location (e.g. Grün et al. 2008; Rasinger et al. 2007; Ricci 2010). These services are even powerful enough to impact the overall travel experience by influencing the sense of tourism, the nature of travel, and the value of places or locations (Gretzel et al. 2008). Accordingly, DMO’s and other providers of eTourism websites (e.g. hotels, skiing areas) have to consider, whether offering their content on mobile devices and developing a specially adapted mobile website next to their desktop version is worth the (quite substantial) investment.

Mobile devices are considered very personal and users are affected by a variety of parameters such as the attractiveness of the device and the applications, the brand, their prior experiences, and the context of use. So far, research focused on the general characteristics of mobile devices (e.g. screen size) and their implications on user experience (Raptis et al. 2013), on the examination of different uses of smartphones and their impact on travel experiences (Wang and Fesenmaier 2013), on the benefits of responsive web design (Gardner 2011), or on the challenges and benefits of bringing web applications to mobile devices (Wessels et al. 2011).

This paper presents an initial step towards understanding the impact of a responsively designed touristic mobile information website on perceived usability, user experience, intuitiveness, and the overall attractiveness.

2 Theoretical Background

2.1 Responsive Web Design

Essentially, Responsive Web Design (RWD) is a methodology introduced to help realizing the vision of a “One Web” (Gardner 2011). Hence, RWD aims to combine the capabilities of HTML5 and CSS3 with a new design paradigm for website architectures, which are able to flexibly adapt to different screen sizes. Marcotte (2011) emphasises this need for an answer to an increasing number of diverse mobile devices and a shifting in user behaviour: “Rather than creating disconnected designs, each tailored to a particular device or browser, we should instead treat them as facets of the same experience.” (p. 8).

Websites without optimizations for mobile devices simply shrink the website to fit the viewable area. This method requires the user to zoom into the website (using touch) in order to read the content properly. A RWD approach alters the layout of the website based upon the viewport of the device, transforming static websites into responsive, adjustable, and fluid layouts, which are much more flexible in handling elements and automatically rearrange them accordingly (Bohyun 2013). RWD changes the way of web development, forcing content providers and developers to consider carefully, what is essential about their content. Therefore, this approach starts with providing minimal services and content in an effective way on the smallest portable device. Afterwards, functionalities and components are added to match devices with larger screen dimensions and different sets of input/output devices. With the lack of proper guidelines, categories are used, which have been derived from an examination of typical device characteristics (Fox 2012). Champeon (2013) summarized this approach under the term Progressive enhancement, encouraging designers to emphasise accessibility, semantic HTML mark-up, external style sheets and scripting technologies. Marcotte (2011) summarizes RWD as a composition of three distinct parts: (1) a flexible grid, (2) flexible images or rather images that work in a flexible context, and (3) media queries to optimize the design for different viewing contexts (devices), and spot-fix bugs that occur at different resolution ranges. Ultimately, RWD effectively adjusts the content and layout to the context of the device and ensures that users have a better and richer viewing experience (Gardner 2011).

2.2 Usability and Mobile Devices

Following Jacob Nielsen (2012), the term usability defines an attribute with qualitative character that assesses the ease-of-use of user interfaces. Usability refers to methods for enhancing this attribute during the design process phase, spreading onto five quality components: learnability, efficiency, memorability, errors, satisfaction, and utility. Usability directly refers to the design’s functionality and investigates, whether the system’s functions meet the user needs.

Within ISO 9241-11, Usability is defined as “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” In more detail, effectiveness covers the accuracy and completeness with which users achieve goals, efficiency evaluates all resources spent in achieving these goals, and satisfaction explains a user’s comfort with and positive attitude towards the use of the system. Although usability provides a valid basis to describe a certain fit of a technology and a user, this three dimensions—effectiveness, efficiency, and satisfaction—limit this approach down to a very functional ‚way-of-doing-things’. Usability therefore stands as a basic evaluation criterion of a technical system (Brau and Sarodnick 2006).

The term “mobile” is traditionally associated with on-the-move, dynamic and portable, while mobile devices are considered personal, portable, and immediate (Wessels et al. 2011). With the appearance of smartphones, an increasing number of users are accessing the mobile Internet via their phone on-the-go, leaving more stationary and familiar settings, like at home or at work (Church and Oliver 2011). Nielsen and Budiu (2013) compared conversion rates when studying e-commerce websites, defining them as “the percentage of visiting users who end up taking a desired action, observing differences, depending on the used device”. Results showed that desktop computers have a 3.5 % rate compared to mobile phones with only 1.4 %. Two possible explanations evolved: (1) Mobile user experience must be horrible, as mobile sales could be 2.5 times higher if mobile websites would be as easy to use as desktop sites, and (2) it is assumed that there is no commitment to invest in mobile design because mobile users do not account for very much business.

2.3 Attractiveness and Intuitiveness in User Experience

An experience that is created when using a system has also implications for certain non-functional aspects within a user interaction (Hassenzahl et al. 2009). Nevertheless, usability should not be considered less important, but should be put into perspective towards a good, or even outstanding, user experience. Within ISO 9241-210, this focus on the individual and its interaction with different objects has been expressed as followed: “A person’s perceptions and responses that result from the use or anticipated use of a product, system or service.” The scope of the term “user experience” is applicable to all objects (tools, entertainment services, knowledge systems, and websites) a person can interact with through a user interface. Studies show that the attractiveness of a product does play an important role in perceiving usability (Chawda et al. 2005). In the field of mobile computing, attractiveness does have the highest influence on usability ratings, followed by effectiveness and efficiency (Quinn and Tran 2010), implying that an attractive phone could have a high usability rating, even when scoring low on either effectiveness or efficiency.

While intuitive interaction lacks a solid best-practice definition and implementation (Ingram et al. 2012) new technologies and interaction gestures (or patterns) provide users with new freedoms, when interacting with user interfaces. This enables developers to design interfaces and create emotions, harvesting on already learned interaction gestures (Tanimura and Ueno 2013). Responsive design and progressive enhancement can be seen here as a natural extension, acknowledging the change in user behaviour. Intuitive interaction is defined as a non-challenging cognitive process and can be narrowed down to mainly information-based activities within a specific context of a task, goal, user, environment, and technical system (Hurtienne et al. 2006). A technical system may be classified as intuitive, whenever an either natural or non-conscious utilization with (or without) a user’s pre-experience leads to an effective interaction (Diefenbach and Ullrich 2011). When designing website experiences for mobile devices through RWD, the interface should therefore not only adhere towards responsive design principles, but should also be designed to emphasise intuitive interaction patterns.

3 Methodology

A user experience experiment was conducted to measure the influence and effect of two different RWD approaches on users’ perceived usability, user experience, intuitiveness, and attractiveness. Two different websites of touristic organizations were chosen: one adhering strictly to RWD principles (Website A, regional tourism marketing) and one following a rather classic approach, providing a mobile website that stays closer to its desktop version (Website B, local DMO). Both offer desktop and mobile versions. The experiment comprised two sessions, where users performed a series of tasks on a desktop computer and a smartphone. Device-wise an Apple iPhone 5s (iOS 7.1.1) and a desktop PC running Microsoft Windows 7 were chosen.

20 persons (14 male, 6 female, age between 16 and 29) participated in the experiment. All participants had at least a basic understanding of using smartphones and the internet in order to cope with the required tasks and all owned a smartphone. As only web browsing was observed, a possible unfamiliarity of Android users with iOS’ look and feel can be assumed not to have any effect in the experiment. The following research hypotheses have been formulated:

  • H1: There is a general difference between the desktop and the smartphone version of the two websites regarding their perceived usability.

  • H2: A stricter implementation of RWD will have a positive effect on the perceived usability of the participants.

  • H3: A stricter implementation of RWD will have a positive effect on the user experience (perceived attractiveness and intuitiveness) of the participants.

3.1 Procedure

The experiment was split in two sessions, one session on the desktop computer and one on the smartphone. In the introduction phase, the participants were introduced to the topic and the setting of the experiment was described. Within the experiment, participants had to accomplish five tasks: three information-seeking and two function-based. All five were classified according to their levels of difficulty (easy, medium, difficult) and the degree of scrolling (easy, medium, heavy). The latter was used to measure its effects on effectiveness and efficiency. None of the participants had any previous experiences with the selected websites (Raptis et al. 2013).

  • Task 1: Subscribe to the newsletter of the website (easy, light scrolling).

  • Task 2: Inform yourself about the Aqua Dome. Please note down the address and phone number (easy, light scrolling).

  • Task 3: Inform yourself about the Hiking Tours in Tirol—“Adlerweg”. Inform yourself about the Hiking Tours in Ötztal—“Ötztal-Trek”. Please note down, how much elevation/how many kilometres the tour comprises (medium, light scrolling).

  • Task 4: Inform yourself about the National Parks. What is the duration in hours of the hiking tour to the Trelebitschsee/Frischmannhütte in the National Park “Hohe Tauern”? (difficult, medium scrolling).

  • Task 5: Please book a vacation using your own criteria on the website, using a budget of 1,500 Euros. Define your trip first using the following attributes: Date of Arrival/Departure, City/Village, Category, and Number of adults/children (difficult, heavy scrolling).

The aim was to discover the content and functionalities of the two different providers and the two versions. Tasks were executed in an A/B test-setting (also called split testing) comparing two versions of a web page (Website A and B) to identify which one performs better (Brau and Sarodnick 2006). Users worked randomly with either the first or the second version of websites and were separated into Group A and Group B (Sauro and Lewis 2012). Predefined criteria were measured in order to compare the results of the two tested groups. After 2 weeks, participants concluded the experiment, working with the other two versions of the websites. During the experiment, participants were invited to share their thoughts thinking aloud. User screen activity and audio was recorded. After finishing all tasks, participants completed a post-test questionnaire.

3.2 Applied Measures

In order to collect a multi-dimensional rating of the participants and to assess the performance of the users the following measures were used:

Perceived Usability

The participants evaluated each version of the websites via the System Usability Scale questionnaire (SUS) (Brooke 1996). SUS comprises ten questions (on a 5 or 7-point Likert Scale) and calculates a value between 0 and 100 (100 = perfect usability).

Perceived Satisfaction and Promotion

The participants evaluated each version of the websites using the Net Promoter© Score (NPS) questionnaire (Reichheld 2003). NPS comprises one question (How likely would you recommend X to a friend?) on a 1–10 scale. Users are then categorized as Promoters (score 9–10) and Detractors (score 0–6). NPS is calculated as the % of Promoters minus the % or Detractors (score between: −100 and 100). In order to predict the NPS out of SUS scores, Lewis (2012) used over 2,000 pairs of scores in order to establish a regression equation. For a more simplified version of this equation (LTR—Likelihood to recommend) the following formula can be used:

$$ LTR=\frac{SUS}{10} $$

Perceived Intuitiveness

The intuitiveness of the different versions of the websites was measured with the INTUI questionnaire. INTUI offers a differentiated interpretation of evaluation results, measuring intuitiveness on four dimensions: (1) Effortlessness (E), (2) Gut feeling (G), (3) Magical Experience (X), and (4) Verbalizability (V) (Ullrich and Diefenbach 2010).

Perceived Attractiveness

The AttrakDiff2 questionnaire (www.attrakdiff.de) was used in order to measure the perceived attractiveness of the different website versions and to evaluate their interactivity. The questionnaire records the perceived pragmatic quality, the hedonic quality and the attractiveness of an interactive product (Hassenzahl et al. 2003).

4 Results

Perceived Usability

The following mean SUS scores, standard deviations, and confidence intervals (α = 5 %) were measured for all four versions of websites. As the sample size of the usability study was smaller than 25 participants, the geometric mean was used to estimate the mean values of the different versions (Sauro and Lewis 2012).

Comparing the SUS scores, a measurable difference between smartphone versions and desktop versions is identifiable—desktop versions were rated significantly higher than their smartphone counterparts: Website B (less responsive) was rated higher by 14.44 (Mean-Desk: 77.35, Mean-Smart: 62.91) compared to Website A (more responsive) with 9.52 (Mean-Desk: 73.58, Mean-Smart: 64.06). Applying the adjective rating scale from Bangor et al. (2009) both desktop website scores were rated “Good”, while the smartphone version scores were between “OK” and “Good”. In relation to a comparative SUS study crossing different domains, desktop web interfaces average on 68.2 while mobile phones average on 65.9.

Net Promoter Score

The best NPS score is −5 (SD = 1.94, mean 7.1) by Website A-Desk. Website A-Smart reached −40 (SD = 2.52, mean 6.0). Website B-Smart reached −45 (SD = 2.82, mean 5.4), and the desktop version −10 (SD = 1.72, mean 7.3). The general average NPS for companies and services is about 5–10. Results below zero imply more detractors than promoters for the tested product or service. Our results show a significant difference between desktop and smartphone versions. A NPS for smartphone versions with −40 for Website A-Smart (Mean-SUS: 58) and −45 for Website B-Smart (Mean-SUS: 55) implies that there are 40 and 45 % more detractors than promoters (Mean SUS: Website-A: 82.5; Website-B: 84.2), while desktop versions seem more neutral with NPS scores of −5 for Website A-Desk and −10 for Website B-Desk.

INTUI

Results show a significant difference within Effortlessness (E). Both desktop versions were perceived as easier to use (Website A: 5.37; Website B: 5.47; higher means better). This result is in line with SUS results where both desktop versions were also rated higher. Gut feeling (G) showed that all versions are rather mind-driven. Website A-Smart (3.67) was the one perceived as the most feelings-driven version (Website B-Smart: 3.20), supporting the theory that a higher compliance with guidelines of mobile usability and RWD leads to an experience that feels more intuitive. Magical Experience (X) showed that Website B-Desk (4.44) and B-Smart (3.83) are perceived as more magical compared to Website A (Desk: 3.64; Smart: 3.40) due to a more picture-rich presentation and design. Verbalizability (V) showed no difference between all versions, meaning that all operating steps were quite easy to remember and verbalizable (mean 5.06). Interestingly, Website A-Smart achieved higher mean values of E, G, X, V than its desktop version and also performed better than Website B-Smart. The observation is another indication that the stricter approach to responsive design guidelines leads to an easier use of the system. The overall intuitiveness of the websites was measured with the component of Global Intuitiveness (INT) with Website B-Desk scoring best (4.70), followed by Website A-Desk (4.63), Website A-Smart (4.33), and Website B-Smart (4.11), 1 being not and 7 being very intuitive.

AttrackDiff2

Within the portfolio-view, the relative characteristics of the two dimensions of pragmatic quality (PQ) and hedonic quality (HQ) are analysed and combined, resulting in a special character for each tested product (see Fig. 1).

Fig. 1
figure 1

Portfolio with average values of the dimensions PQ and HQ and the respective confidence rectangles—Smartphone (left) and Desktop (right)

Full size image

Website A-Smart was rated generally neutral, lacking on hedonic quality, and being stronger on the pragmatic side, possibly due to a stronger adherence of responsiveness, lacking pictures and focusing more on the content. Nevertheless users still seem to be stimulated by the site. Website B-Smart shows a higher evaluation of hedonic quality, due to a richer use of graphical components and pictures, but no improvement towards the pragmatic quality. Both desktop websites improve their score significantly on both dimensions, with Website B-Desk achieving the highest score, possibly again due to a richer graphical design and look and feel, implying a strong emotionally stimulating factor of this website. The diagram of means for the different components was measured to show the critical dimensions of the tested websites and visualize where the websites differ from each other (see Fig. 2).

Fig. 2
figure 2

Mean values of the four AttrakDiff2 dimensions for the two websites—Smartphone (left) and Desktop (right)

Full size image

Website A-Smart is rated average, or rather ordinary, on all dimensions: Pragmatic Quality (PQ), Hedonic Quality-Identity (HQ-I), Hedonic Quality-Stimulation (HQ-S), and Attractiveness (ATT). For Website B-Smart, HQ-S is evaluated significantly better, than for Website A-Smart. Both desktop versions improve in all dimensions, especially on attractiveness, with Website B being rated higher on HQ-I, HQ-S and ATT. The main message of the diagram of means is similar to the results of the portfolio-view: Website B achieved slightly better results in the categories of HQ-I, HQ-S and ATT. The component of PQ showed no differences between the two websites when used on either version. The general trend that the smartphone versions were rated lower was underpinned by comparing the average values. The components of ATT and HQ-S underline that Website B was perceived as more attractive and more creative.

4.1 Statistical Analysis

A within-subjects ANOVA was performed for the measures of perceived usability (SUS and INTUI-E). The components of overall Intuitiveness (INT) and Magical Experience (X) were also analysed using a Kruskal-Wallis test. The effect of the participants’ operating system on SUS was measured with a two-tailed T-Test. Each of the following analyses ran through the required pre-tests of the chosen model (Shapiro-Wilk and Mauchly test).

  • H1: There is a general difference between the desktop and the smartphone version of the two websites regarding their perceived usability.

An analysis was conducted to investigate the effect of all tested website versions on perceived usability. The variables used were the different SUS scores for all versions. Results for the first within-subjects ANOVA showed that there is a significant effect for the version of the websites on perceived usability, Wilks’ Lambda = 0.62, F(3,17) = 3.48, p = 0.039, partial Eta2 = 0.38 and compliance with Mauchly’s Sphericity p = 0.662. A post-hoc analysis (Fischer’s Least Significant Difference Test = LSD) was conducted. The pairwise comparisons indicated, that there is a significant difference between Website B-Smart and Website B-Desk (p = 0.022), but no difference between Website A-Smart and Website A-Desk (p = 0.158). As a first conclusion, H1 cannot be substantially proven, showing a significant difference on perceived usability between Websites B, but none between Websites A. No significant difference between smartphone versions (p = 0.829) can be found, hence H2 has to be rejected.

  • H2: A stricter implementation of RWD will have a positive effect on the perceived usability of the participants.

In order to investigate H1 and H2 from another perspective, a second analysis was conducted. As Ulrich and Diefenbach (2010) mentioned that Effortlessness (E) has the strongest relation to classical usability metrics, related effects of each website version on this component were analysed. Results of the within-subjects ANOVA show a significant effect of the version of the website on usability, when compared with the component of Effortlessness: Wilks’ Lambda = 0.46, F(3,17) = 6.75, p = 0.003, partial Eta2 = 0.54 and compliance with Mauchly’s Sphericity p = 0.296. Post-hoc analysis (Bonferroni) indicated a significant difference between Website B-Desk and Website B-Smart (p = 0.006) and no significant difference between Website A-Desk and Website A-Smart (p = 1.000). Again the post-hoc analysis indicated that H2 cannot be substantially proven. In addition, no significant differences between the both smartphone versions could be found.

  • H3: A stricter implementation of RWD will have a positive effect on the user experience (perceived attractiveness and intuitiveness) of the participants.

The component of overall Intuitiveness (INT) was used to measure the effect of each website version on global intuitive interaction. As the data cannot assumed to be normally distributed (Shapiro-Wilk: p < 0.05) a second Kruskal-Wallis test was applied. The test resulted in no significant difference between the versions (p = 0.256). A second analysis using Magical Experience (X) was conducted to approach the assumption from another perspective. As the data for Website B cannot assumed to be normally distributed (Shapiro-Wilk: p < 0.05) a Kruskal-Wallis test was used. The analysis showed no significant differences between the versions (p = 0.065). This implies that the third hypothesis (H3) had to be rejected. In addition, AttrakDiff2 results indicate that differences in terms of user experience were not relevant enough to be significant.

Correlations Between SUS and NPS Scores

A Spearman-test and Pearson-test have been applied to measure correlations between SUS and NPS. Sauro and Lewis (2010) compared the data of 146 users who were asked to complete both a SUS for perceived usability and a NPS for customer loyalty and found a significant correlation (r = 0.61, p > 0.001) between the two variables. Hence, promoters have an average SUS of 82, compared to detractors with an average SUS of 67. In our study these correlations for Website A-Smart (Mean SUS—Prom.: 82.5; Detrac.: 58), Website A-Desk (Prom.: 88; Detrac.: 62.5), Website B-Smart (Prom.: 84.2; Detrac.: 55), and Website B-Desk (Prom. 92.5; Detrac.: 62.5) were significant at an alpha level of 0.01. The outcome strongly supports above assumptions, with detractors and promoters varying at the same level. The correlation underlines that participants who face problems in the task execution and express those problems through SUS are transferring their collected impressions when evaluating the NPS.

5 Discussion

“Does a higher (=stricter) degree of responsive web design enhance the usability (perceived and measured) of a mobile information website?” Website A-Smart (stronger compliance to RWD) achieved better results in terms of Perceived Usability and Intuitiveness. However, the differences were only significant for Websites B and not for Websites A, leading to contradictory results. Additional research needs to be conducted to further investigate the impact of RWD on perceived usability. The general differences between the two smartphone versions showed that Website A was marginally ahead in terms of SUS scores, NPS, and overall intuition, the overall Attractiveness of Website B-Smart was slightly better. Our conclusion here is that RWD is a possible way to enhance perceived usability, but that a noticeable trade-off towards being too pragmatic (or boring) is likely to be created and smartphone users will be negatively affected by it.

“How does a stricter approach of responsive web design influence the user experience?” Our data analysis did not discover a significant difference between the two tested websites (H3 was rejected). However, the participants indicated that Website A-Smart was perceived as easier to use and that single operating steps were easier to remember. On the other hand, again Website B-Smart was rated higher in terms of Magical Experience, meaning that the interaction with the website was perceived as extraordinary and fascinating. This is due to the website’s better use of mainly optical features, such as good design, number of pictures and overall attractiveness. Participants who preferred Website A-Smart answered more in a direction of logical reasons, like clear structure, ease-of-use and well-designed content. Concluding, users who favour websites with a focus on aesthetics and design will be negatively influenced by stricter approaches to RWD. The challenge of successfully implementing a mobile website will be to find an adequate balance between aesthetical aspects and efforts to optimize the usability and compatibility according to a device’s requirements. This should not imply that efforts to optimize for mobile interfaces automatically lead to negative user experiences, but that all efforts in the process of reducing functions and design elements should be considered very carefully, as a direct negative influence towards promoting the website was identified.

Our user experience study illustrates that there are still massive differences in usability and user experience scores between desktop computers and smartphones. While both websites had different approaches to the implementation of RWD, the one that had a more compliant adherence to the guidelines achieved better overall results, but failed to attract and be recommended. Implementing guidelines alone does not lead to better results of usability and user experience. They have to be prioritized by criticality in order to achieve a higher relation to the effects on performance and/or satisfaction. As a result, developers should select and integrate RWD guidelines based on their content and level of importance to their strategic goals (Hart et al. 2008).

6 Conclusion

The impact of responsive web design on user experience needs to be addressed from multiple viewpoints. The outcome of this study indicates that strong reductions on visual features can have negative consequences for the user experience, which impacts users of touristic mobile website, who expect to encounter very picture rich and graphical designs. On the other hand, such reductions can also lead to improved usability and increased overall satisfaction. The discovered correlation between SUS and NPS scores clearly illustrates the connection between usability and user experience and their influence on promotion and detraction of a mobile information service, being a critical success factor especially for DMO’s and tourism service providers. As Reichheld (2003) stated, no one is going to recommend a product or service without really liking it. Consequently, the increasingly growing mobile user group needs to be addressed with greatest care, employing a user-centred design process that relies on established models and guidelines. Ultimately, users will decide on the success or failure of such implementations. However, magical experiences and attractiveness should not be neglected—rainbows and unicorns included.