1 Introduction

Although a considerable amount of work has been undertaken from a usability perspective in the design of mobile and wearable devices [1–4], the established emphasis has been on issues such as ergonomics, multimodal interaction, haptics, personalisation, navigation, as well as novel interaction paradigms. Consequently, such work has frequently ignored the initial user experiences with the device—the out-of-box-experience (OOBE)— which in many cases can be crucial to user acceptance of the technology.

Whilst there has been an increasing recognition of the importance of the OOBE to the consumer experience, research in this area has prevalently examined issues such as ease of use, the establishing of appropriate factorisations of the OOBE, as well as the creation of appropriate evaluation methodologies [5–8]. Moreover, while there have been indications that the OOBE may suffer when the user contact extends across several companies [9] an important question, namely, how is the OOBE affected when the experience itself spans several devices, is yet to be answered though.

In this paper, we address this issue and explore the user OOBE for two mobile devices: a Personal Digital Assistant (PDA) and a wearable Head Mounted Device (HMD), both of which are ultimately interconnected to give the user a mobile, wearable, computing experience. Accordingly, the structure of the paper is as follows: Sects. 2 and 3 describe general usability issues pertaining to PDAs and HMDs, respectively, whilst Sect. 4 gives an overview of the main themes of OOBE research. Sect. 5 details the methodology followed by our experiments, while results are presented and analysed in Sect. 6. Lastly, conclusions are drawn and possibilities for future work identified in Sect. 7.

2 Personal digital assistants

Improvements in technology, especially in wireless networking, have pushed the barriers of anywhere/anytime information access. Portable information access raises the need for portable information access devices, such as communicator devices and personal digital assistants, which promise to supplant the desktop computer as ubiquitous technology on campuses and in business [10], with Gartner Research predicts a 260% increase in the unit sales from 9.39 million units in 2000, to 33.7 million units in 2004 [11].

Personal Digital Assistants inherit human–computer interaction and ergonomic related issues, such as small screen size, slow input facilities, low bandwidth, small storage capacity, limited battery lifetime and slow computer processor unit speed, which are all possible obstacles to the success of mobile and pervasive computing objectives [1, 12].

It is not surprising then that PDA research is mainly involved in overcoming such barriers. Thus, for instance, Jones et al. [13] studied the effect that screen sizes have on web-browsing related tasks, and their results showed that users with small screens followed hyper-links less frequently than those with a larger display unit. In related work, MacKay [3] looked into the web content adaptation for small-screened devices using a gateway/transcoder. The proposed gateway displays the web page designed for a large screen on a smaller one by reducing the web page in scale to fit the screen. The users of the gateway can focus on any part of the screen to interact with it or just to navigate, helping the user to create a mental model of the site (s)he is visiting and feel in control. From a different perspective, Waycott and Kukulska-Hulme [4] evaluated student experiences using PDAs when employed as tools for reading and learning, and their findings showed that due to the aforementioned problems typical of PDAs, it was difficult to read and interact with documents using such devices. A fresh approach is the one proposed by Siemens Corporate Research, who put forward a framework that accommodates ubiquitous multimedia access using small-screened devices. This allows PDA users to access rich multimedia content and services without having to shrink or tailor the content to match the capabilities of the PDA device. This is achieved by the collaboration of the PDA with its surrounding electronic appliances (e.g. TV, mobile phone), under the coordination of a Smart Server [14].

3 Head mounted displays

Head-mounted displays are a sub-set of wearable computer technology, which aim to allow hands free access to computer functionality. Users’ reasons for hands free access to computing devices is often varied, and range from individuals with a restrictive physical disability [15] to individuals working in dangerous or hazardous conditions [16]. Integration of wearable mobile devices with network technology, touch pen, speech recognition inputs, interactive glove or face-mounted devices (as in the case of Xybernaut’s Mobile Assistant) allow extremely adaptable mobile solutions. Devices, such as head-mounted displays, have often been considered synonymous with virtual reality development, however, due to falling cost and improved technology, head-mounted displays devices are becoming more commercially available and have recently gained commercial importance for high street companies such as Olympus and Sony.

The head-mounted display is made of two canonical displays, and usually consists of two liquid crystal or cathode-ray tube display screens that are either mounted on a helmet or glasses-frame structure.

There are several attributes that affect the usability of the head-mounted displays. Head-mounted displays can be either binocular, showing the same image to both eyes, or stereoscopic in nature, showing different images to each eye. The choice between binocular or stereoscopic depends on whether three-dimensional interaction or presentation is required. Whilst head-mounted displays use a range of display resolutions, it is important to note, however, that a trade off exists between the resolution used and the field of view, which in turn impacts the perceived level of experienced immersion. A low field of view decreases the experienced level of user immersion, yet a higher field of view involves spreading the available pixels, which can cause distortion on the picture. Finally, ergonomic and usability factors vary considerably between different devices. Issues such as display size, weight and adjustability of physical and visual settings all affect the usability of a particular head-mounted display for any specific task [17].

Although there is now a wide range of head-mounted displays, there are several drawbacks that prevent their everyday popularity. Thus, the current high cost of the head-mounted displays that display both high resolution and wide field of view is a major factor. The large and encumbering size is also an important factor for users, especially those of cathode-ray tube based displays [18]. Moreover, the visual limitation within the real world and reduced interactions with colleagues are also possible reasons that prevent head-mounted displays from regular everyday popularity. Lastly, other factors, such as hygiene and weight, also have possible unknown long-term medical implications on the supporting muscles and, indeed, even on the eyes.

A number of research studies thus exist looking at the symptoms related to head-mounted display usage, such as nausea [19], dizziness [20], headaches [21] and eyestrain [22]. From a different perspective, Geelhoed et al. [23] investigated the comfort level of various tasks, such as text reading and video watching, on two different head-mounted displays, identifying that tasks requiring more long-term attention, such as watching video, causes a greater level of discomfort to the user.

Despite the computational costs and usability drawbacks of the head-mounted displays, they are used widely in active research, ranging from virtual environments [24] to wearable Internet applications [25]. However, to the best of our knowledge, the OOBE associated with HMDs is an insufficiently explored research area; moreover, bearing in mind the inter-operability and inter-connectivity of devices, it is essential that the OOBE of HMDs when connected to other devices, such as PDAs, is examined, and this is what we endeavour to do in the present paper.

4 The out of box experience

An integral part of the human–computer interaction is represented by the initial user contact and experiences with the computers—OOBE—which is of a great strategic and marketing importance. Accordingly, the OOBE has been acknowledged as such by most major players in the industry, through studying it within the context of in-house research groups [7] or cross-company consortia [26], and by endeavouring to ensure that products are shipped with the capability and know-how that will enhance the user’s/buyer’s OOBE [27].

The challenges faced by designers of the OOBE are numerous: these range from ensuring a consistent initial user experience across a range of possibly interconnected/interdependent devices [9], to establishing the essential components of the OOBE [28], and indeed, towards formulating a generic approach for the OOBE design itself [7, 8, 29].

The importance of the OOBE has also been highlighted in a study (EoURoundtable 2000), which examined core areas of user dissatisfaction, based on data gathered from usability tests, tech centre support calls, and user research from 26 companies, including major industry players such as Intel, Microsoft, IBM, Dell, Siemens and Hitachi. According to the findings of the report, there were eight main causes behind user dissatisfaction (Table 1), with OOBE being the only non-technical one.

Table 1 Main causes of ease of use dissatisfaction (from EoURoundtable 2000)
Full size table

Further examination of user experiences [26] showed that the most frequent causes for discontent with the OOBE were

  • PC initial setup time;

  • Too many physical cables (connectors, adaptors);

  • An overwhelming amount of documentation, and

  • Confusing content of user manuals.

Lastly, OOBE research is ongoing and currently examining the experience impact of mobile and wireless devices. The findings on wireless devices show that the setup of such equipment requires that the user is familiar with networking and computer hardware, and, because of this, it is frequently an intimidating and frustrating process for most users [6]. Moreover, research has also highlighted that the notion of a singular, device-specific, OOBE is potentially flawed—the initial user experience increasingly concerns the configuration of more than one device, and the consequent interaction between the different device-specific and company-specific OOBEs can lead to less than desirable outcomes [9]. Indeed, in our work, the details of which we give in the next section, we have sought to examine the OOBE when multiple devices, from different companies, are set-up and subsequently interconnected for intial use.

5 Experiment description

5.1 Participants

Our study involved 18 participants, nine male and nine female, who were aged between 23 and 37 and were taken from a range of different nationalities and backgrounds—students, clerical and academic staff, as well as white and blue collar workers. All participants spoke English well enough to be educated in this medium, whilst the average duration of participant computer usage was 7 years and 1 month. All participants had not previously used the type of devices employed in our OOBE experiment.

Using the taxonomy of McMurtrey [28] the users were divided into three categories: Novice, Familiar and Experienced. Each user was assigned to one of these categories according to the answers that (s)he gave to a set of seven questions in the “Background” section (Table 2) of the OOBE questionnaire used in our experiments (the questionnaire is described in detail in Sect. 5.4). Accordingly, users who answered “yes” to 0–2 of the questions were categorised as Novice, those who answered “yes” to 3–4 of the questions were categorised as Novice, those who answered “yes” to 3–4 of the questions were deemed to be Familiar, whilst those who answered “yes” to 5–6 questions were assigned to the Experienced category. The split of users according to gender and user category is given in Table 3, where, as can be seen, the participants in our experiments turned out to be evenly distributed in terms of their computer expertise.

Table 2 User profile questionnaire
Full size table
Table 3 Participant breakdown according to gender and type of user
Full size table

5.2 Experimental variables

Three experimental variables were examined in our study—these were type of user, type of device and gender. Type of user is an important factor which has been identified in the OOBE context [8], whereas the influence of device type was looked at in order to gauge the effect, if any, on the OOBE. The two types of devices used in our experiments were a PDA and an HMD (whose technical characteristics are given in Sect. 5.3), whilst the possible impact of gender on the OOBE was also examined, in order to see the extent to which possible customisation of the OOBE can reach.

Data were analysed with the Statistical Package for the Social Sciences (SPSS) for Windows version (release 11.5). An ANalysis Of VAriance (ANOVA), suitable to test the significant differences of three or more categories, and t-test, suitable to identify the differences between two categories [30], were applied to analyse the participants’ responses. A significance level of p<0.05 was adopted for the study.

5.3 Experimental material

In our experiments, participants interacted with two devices. The first was an Olympus Eye-Trek FMD 200 head-mounted display, which uses two liquid crystal displays and allows a greater autonomy of movement than a generic computer monitor. Each one of the displays contains 180.000 pixels and the viewing angle is 30.0° horizontal, 27.0° vertical. It supports PAL (Phase Alternating Line) format and display weight is 85 g (Fig. 1).

Fig. 1
figure 1

Head Mounted Display Device

Full size image

The other device was a Hewlett-Packard iPAQ 5450 personal digital assistant with 16-bit touch sensitive transflective thin film translator liquid crystal display that supports 65,536 colour. The display pixel pitch of the device is 0.24 mm and its viewable image size is 2.26 inch wide and 3.02 inch tall. It runs the Microsoft Windows for Pocket PC 2002 operating system on Intel 400 Mhz XSCALE processor and allows the user complete mobility. By default it contains 64 MB standard memory and 48 MB internal flash read only memory. Additionally, in the course of this experiment, a 128 MB Secure Digital memory card was also used (Fig. 2), while the interface between the HMD and the PDA was provided by a Lifeview FlyJacket i3800. The FlyJacket has its own internal rechargeable battery and provides VGA, S-Video and Composite output from the PDA device. It supports full motion video previewing up to 30 frames per second and 1024×768 VGA resolution.

Fig. 2
figure 2

Personal Digital Assistant with Lifeview Flyjacket i3800 attached

Full size image

5.4 Experimental process

Our experiments followed the User-Centred Design (UCD) process [7] in order to appraise the OOBE. Accordingly, user OOBE was broken down into eight categories, with the user having to accomplish a set of five tasks for each of the categories identified (Appendix 1). The OOBE categories were

5.4.1 C1: Packaging

This category measures the users’ first impressions regarding the packaging of the equipment. Ideally, a product should be packaged in a way that allows the user to transport it to a target location easily and safely, and clearly identify the contents of each box. Thus, in this category the participants were asked to check the weight of the packages, to identify the content of the package based on their labels and finally to check whether or not it would be easy to transport them.

5.4.2 C2: Unpacking

This category concerns itself with the efficiency and the easiness of the users’ unpacking experience. The aim is for the package to be ready for setup quickly and easily. In this category participants were asked to understand the interior organisation of the package, do an inventory check on the content using the manual and finally to identify the related components (i.e. cables, adaptors and connectors).

5.4.3 C3: Setup

The easiness of physical arrangement and assembly of components, as well as intuitiveness of the related instructions provided are examined here. The aim is to prepare all components to use as quickly as possible and leave no opportunity for mistakes. In this category, the participants are asked to understand the physical arrangement of the components and assemble them.

5.4.4 C4: Power on

The design of the power-on experience is looked at here: this includes issues such as feedback that setup was successful, rewards, together with elements such as thank-you and welcome messages. The aim is to immediately verify that setup or assembly was done properly and everything is working correctly. In this category, the participants were asked to read the appropriate section of the user manuals and turn on the devices to observe the responses according to their expectations.

5.4.5 C5: Configuration

This class of questions measures the post-setup configuration experience. The aim of the configuration element is to be as automatic and transparent as possible, with minimal user interaction. Due to the substantial difference between device types and their respective configuration methods, two separate sets of tasks were prepared. For the PDA, the participants were asked to agree with licence terms and install ActiveSync in order to subsequently synchronise the Pocket PC with a desktop computer. For the HMD, the participants were asked to link the HMD with the PDA and unlock the HMD device using a default password.

5.4.6 C6: Initial Use

This category measures easy accessibility of product features and capabilities. The main aim is to reaffirm that users’ product decision was wise and give them confidence that they will be able to use the product to its fullest capabilities. A similar approach to the Configuration category is used due to the feature differences of the devices concerned. For the PDA, participants were asked to personalise and synchronise the device with a desktop computer, and set the security settings so no one else can access their data. For the HMD, participants were asked to personalise the device colour settings and reset the password according to their preference.

5.4.7 C7: Doing work

This class of questions examines how easy it is for the user to do meaningful things that they want to do. The main aim is to make productive use of the device used. Accordingly, for both devices the participants were asked to transfer a video clip file from a desktop computer to the PDA, and play the specific movie on the PDA. Following this, they had to turn on the wireless networking and surf the local network.

5.4.8 C8: Assistance

The availability of multiple assistance sources for every step of the initial experience, such as written instructions, troubleshooters, online support, and user help groups is typically examined here. The aim is to help the user resolve problems and get assistance as quickly as possible. In this category, the participants were asked whether the devices provide real-time help, whether a support website is clearly provided on the users manual, and if the respective manuals’ structure is considered to be user friendly.

User OOBE was assessed via a questionnaire (given in Appendix 2), which the participants had to complete as they went through the tasks. Accordingly, after the users had completed the profile questions, they were asked to indicate, on a five-point Likert scale of 1–5, for each task undertaken, their opinions (ranging from “1— Strongly Disagree” to “5— Strongly Agree”) with respect to a statement regarding the respective task. In its final part, the questionnaire also solicited any open-ended comments that the participants might have wished to make.

Participants were accordingly instructed to work their way through the tasks, and complete the relevant sections of the questionnaire as they went along. To ensure that the participants did not feel under test conditions, it was made clear that their skills/ability to interact with the devices was not being tested (rather it was the case of evaluating initial contact with the devices), and that they consequently should not be concerned if they were unable to complete any of the tasks that the experiment contained. Furthermore, they were told to accomplish the tasks at their own pace, with no pressure being put on the part of the experimenter (the same experimenter was used throughout) for participants to abort/finish early any of the experimental tasks. However, as the duration of the OOBE is one of its key components [26], the overall time in which the experiment was completed was recorded for each user, even though (s)he was unaware that this was being done. Accordingly, the fastest completion time for the experiments was 95 min, whilst the longest was 211 min. The average duration of our experiment was calculated to be 136 min. Lastly, we mention that environmental variables were kept constant for all participants in the experiment, as this took place in the same room of the Brunel University usability laboratory.

A pilot test study of two participants was used to check and validate the questionnaire and the experimental process. Apart from requests for slight rephrasing of questions/statements in order to ease their understanding, this study revealed that one of the initial tasks in the “Doing work” category could not be successfully undertaken due to limitations of PDA memory size and battery power. Accordingly, in the final version of our experiments, this task was replaced by a feasible one (Task T3 of Category C7, as given in Appendix 1).

6 Results

6.1 Device impact on the OOBE

The most important result to come out of our analysis was that there are significant differences in the Configuration and Doing Work categories of the user OOBE, depending on the particular device at hand—PDA or HMD. Accordingly a Wilcoxon test highlighted that in the Configuration category, users found tasks T1–T4 to be significantly easier in the case of the PDA (Fig. 3), whilst in the Doing work category, users responses indicated that all tasks in this class were again significantly easier in the case of the PDA (Fig. 4).

Fig. 3
figure 3

PDA versus HMD—OOBE for the Configuration category

Full size image
Fig. 4
figure 4

PDA versus HMD—OOBE for the Doing Work category

Full size image

This result highlights the difficulties that users found in interconnecting devices, and particularly with wiring and audio/video cable identification, essential components of this goal, as is software installation. Thus, in order to link the HMD with the PDA, users had to attach the Lifeview FlyJacket to the PDA and install the associated device drivers. Whilst users did not encounter particular difficulty in installing the ActiveSync software associated with the PDA, they did so in the case of the FlyJacket device drivers. A similar observation can be made with respect to the interconnection between the PDA and the desktop machine, where users had comparatively little trouble achieving this, whilst the task of connecting the PDA to the HMD was found to be significantly more difficult. This result complements the observation of Fouts [9], who argues that the OOBE, when extended across several companies, can be compromised—in our research we did indeed find that the initial interconnection of different devices proved to be one of the hardest tasks experienced by users, with consequent negative implications on their OOBE.

Our experiments also showed that routine tasks such as file transfers and the running of a typical application (Windows Media Player) was more difficult when users had to accomplish these when viewing the interface on the HMD, as opposed to the simpler case when direct input to the PDA was required. Moreover, for both the Configuration and Doing Work categories of the OOBE, the documentation was found to be more helpful in the case of the PDA. This was in spite of the fact that the PDA did not have one single manual which the users could consult, but rather a collection of booklets, each addressing particular functionalities (e.g. Bluetooth, WiFi, Setup), which was found by some participants to be a source of insecurity and disorientation in finding help. However, in the case of troubleshooting for help in the setup of the two devices (the last task of the Setup category), the fact that the HMD had an explicit troubleshooting section in its manual explains why users found the HMD documentation in this case to be better than the one of the PDA.

Analysis of our results also revealed that the carrying the devices’ boxes (task T1 of the Packaging category) was found to be significantly easier in the case of the HMD, though, which might be a reflection of the fact that, whilst the overall weight of the two boxes was similar, the one containing the PDA was more voluminous. On the other hand, the easiness of identifying the boxes’ respective contents based on the labelling used, was found to be significantly greater in the case of the PDA. This, in our opinion is due to a rather ambiguous exterior label on the HMD box (which depicted a person wearing an HMD whilst also playing a computer game on a console), whilst the PDA box had a clear picture of the device contained therein, unencumbered by secondary connections. A related observation is to be made in the case of repacking (task T5 of the Unpacking category), which was found to be significantly easier in the case of the PDA, where internal compartmentalisation was aided by appropriately moulded plastic diving sheets, as opposed to the case of the HMD which relied on simple, unmoulded, cardboard dividers, whose exact position in the box, once taken out, was far from straight forward to figure out.

Lastly, the OOBE was significantly better in the case of the PDA than that of the HMD for task T4 of the Initial Use category, which concerned itself with initialising security settings for the devices, and for task T1 of the Assistance category. The former result probably is a consequence of the fact that the PDA has a dedicated, streamlined interface for this task, in contrast to the HMD in which this task can only be accomplished with the user interacting via a non-dedicated, multi-functional set of buttons. On the other hand, the latter result comes about due to the PDA running a modified version (Windows CE) of the Windows Operating System, and users’ general familiarity with obtaining help in this environment.

6.2 Impact of user type on the OOBE

The OOBE for the two devices considered by our study is generally unaffected by the particular user type (Novice, Familiar and Experienced)—this is the conclusion of a Kruskal Wallis k-sample test applied to our results. This analysis highlighted that only sporadic tasks (four in all, out of a total of forty) were found to be user-dependent, an observation which probably reflects the care taken by designers to make sure that the OOBE is not perceived differently by different categories of users (Figs. 5, 6). Moreover, this conclusion is further reinforced by the fact that the same analysis also revealed that the length of time taken by users to complete our experiment is again unaffected by the particular user type.

Fig. 5
figure 5

The OOBE for the PDA according to User Types

Full size image
Fig. 6
figure 6

The OOBE for the HMD according to User Types

Full size image

Tasks that were found to be dependent on user-type include, in the case of the PDA, (surprisingly) task T3 of the Power On category and task T5 of the Initial Use category. In the case of the HMD, tasks T1 and T3 for the Initial Use category were found to be user-dependent. The first of these has to do with noticing the blinking light of the PDA, indicating that the device is being powered. We believe that user-type being an influencing factor in this case is probably a reflection of the fact that Novice users in our experiments, whilst noticing the light itself, did not in fact make the connection to it being an indicator of available power supply. The fact that Novice and Intermediate users found transferring files on the Secure Digital memory card of the PDA difficult is, in this case, expected, and most probably due to the technical know-how and navigational skills required to accomplish this particular task.

The last two tasks found to be user-dependent is a possible indication of the rather limited, console-type interface (consisting of four multi-functional buttons) that the HMD provided for their realisation on its control panel. Accordingly, the two tasks focused on switching on the HMD and the setting of appropriate colour/brightness levels, and it is quite likely that Experienced users, given their familiarity with the concept and use of multi-functional interfaces, would have found these tasks significantly easier than the other user categories considered in our study.

6.3 Gender impact on the OOBE

A Mann–Whitney test on our experimental results showed that there were no significant differences between females and males as regards their OOBE for the HMD (Fig. 7). The same analysis revealed that, with the exception of six tasks, the same observation held true for the PDA (Fig. 8). Additionally, no significant differences were found between the lengths of time it took females and males to complete the tasks of our experiments.

Fig. 7
figure 7

Gender impact on the HMD OOBE

Full size image
Fig. 8
figure 8

Gender impact on the PDA OOBE

Full size image

Most of the tasks in which there were significant differences between the PDA OOBE for females and males can be broadly categorised into two main groups: the first mainly covers set-up and interconnection of devices, whilst the second deals with wireless operation. Accordingly, the interconnection between the cradle and the PDA (task T2 of the Configuration category), as well as the synchronisation between the PDA and the desktop, as well as personalising and synchronising the PDA (tasks T2–T4 of the Initial Use category) were found to be harder by female participants. In the second group we find tasks T3 and T4 of the Doing Work category, dealing with configuring the PDA for wireless networking. It is our belief that a possible explanation for these results rests with the diminished spatial ability of females [31], although this is an interesting avenue for future work.

7 Conclusions

This paper examined the OOBE when the experience extends across two interconnected devices—a PDA and a Head Mounted Display. The findings of our empirical study have highlighted several issues: the first is that the device type can have a significant effect on the OOBE, particularly when configuring the devices and trying to perform routine tasks. Moreover, most of the participants in our study found the experience of interconnecting the devices daunting and frustrating.

Whilst our study has a relatively small sample size, our results question the assumption that smart gadgets by themselves have sufficient novelty appeal to ensure a positive user OOBE—this is especially more so when such devices need to be interconnected in order to provide, for instance, the premises for a user to experience wearable computing. This finding would seem to indicate that integration of multiple functionalities on a single device (with the corresponding reduced need for interconnection), coupled with enhanced multi-modal interaction (to possibly compensate for the one-wearable-device-fits-all-functionalities setting), might be the way forward. Whether indeed such a device would have a better OOBE than the scenario studied by us is an interesting question deserving future work.

On a positive note, our findings have also shown that the OOBE is mainly unaffected by user computer experience and gender, which highlights that the OOBE designers of the two devices have made efforts to consider these categories and provide a broadly uniform OOBE. Whilst we attributed the (limited) gender impact to differences in spatial awareness, this in itself is also a worthy opportunity for future exploits.

In concluding, we remark that the OOBE should not be viewed in isolation—indeed if one wishes to have true integration of devices, then this should also happen at all levels of the user experience, beginning with the OOBE. Only by so doing will the claim of building integrated, user-friendly systems ring true.