Keywords

1 Introduction

For years, Virtual Reality (VR) has been experiencing an enormous boost in development, thanks to an increasing amount of technical innovation and it is predicted to continue to grow [28]. Large companies, such as Facebook with the acquisition of Oculus VR, are investing billions in the further development of VR hardware as well as software [3]. This trend is strengthened by the development through further complementary innovations, such as 360-degree video cameras, VR compatible development environments such as Unity [34] or Unreal Engine [35], as well as an unchanged social and entrepreneurial movement to make the knowledge of the world available to all people.

Advancement of VR innovation has impelled a worldwide race to recognize and exploit the next significant market opportunities. VR hardware manufacturers and software engineers are collaborating with specialists of several different areas such as product designers, architects or healthcare practitioners to create the next generation of interactive products and services [40].

Innovation is crucial for businesses nowadays to be successful in the global market. Many companies are trying to gain a competitive advantage towards their competitors by applying new technologies and processes to their engineering design practices. Opportunities in industrial design processes arose because the maturity of VR element technologies has increased greatly [11]. VR opens new possibilities for a faster, cheaper and more effective process and they appear in all stages of product design, such as ergonomics, product life cycle, manufacturing, etc., whereby the biggest potential seem to occur during the early stages of product development. Although computer modelling processes are currently used at different stages of the product life cycle, building a physical, fully functional prototype makes the product development process very expensive and time consuming. New technologies, such as virtual reality applications, are needed to empower the industry with a faster and more efficient way of creating products as they are an ideal tool for simulating prototypes with authentic interactions [33].

In this study, we investigate whether there are differences in the user experience of three different products when users perceive and interact with them. In order to achieve this goal, we conducted a product test that compares the user experience of real products with their representations in virtual reality. The fidelity of the computer-generated experience was still far from that of the real-life ones, thus one of the hidden interests behind the study was whether the level of fidelity that current VR experiences offer would already generate a comparable experience to real-life products in terms of the measurable UX-factors.

2 Background

This paragraph relines the topics User Experience, Virtual Reality and Prototyping and connects them to the background of our study.

2.1 User Experience, Hedonic and Pragmatic Quality

Over the last decades, design professions have become increasingly engaged with UX. Various different attempts at providing theories and UX frameworks have been made [7]. In contrast to early UX research, which was more focused on task efficiency and other pragmatic qualities, user experience nowadays also considers quality attributes like self-actualization [19] and the fact that the consumer experiences hedonic qualities while interacting with a product. In Bruno and Muzzupappa [9], the authors compared the usability of a microwave in virtual reality with the real-life version of the product. The results of this test showed that there was no significant difference between the usability of the same product in VR compared to reality. However, none of the previous research conducted has tested if the user experience of a product is different if it is seen and interacted with in virtual reality compared to reality.

2.2 Virtual Reality

Virtual Reality is the experience to perceive a computer-generated environment as if it is real. It is a powerful tool that provides an advanced immersive medium which can deliver high-impact messages and engage potential consumers [5]. According to Villani et al. [38], the experienced presence of some VR applications might even surpass reality in some occasions, depending on the quality of the application as well as the virtual environment.

Currently, there are examples of VR use which indicate that VR could be a next step in evolving product development processes or that it is already in use for it.

The whole process of assembly planning in product development is a crucial step for many companies. The process that describes how different parts will be put together, function and also how they appear has proven to be a major cost factor for the development of a product [8]. VR technology has proven to be an efficient tool to simulate such advanced interactive product models by also providing different kinds of sensations, such as haptic, auditory and visual [33]. Virtual CAD models allow product designers to import prototypes into virtual environments during early as well as later design stages. As a result, companies are able to perform evaluations based on the virtual simulations instead of building a real product prototype. Especially formal design changes can be incorporated easily using virtual applications. Haptics, collision detection and interactions between objects can be simulated by using natural human motions. Furthermore, the concept to visualize realistic behavior and examine human interactions makes for example virtual assembly prototypes ideal for analyzing assembly related problems [33]. In order to create a reliable product model in VR, the evaluation environment must be able to simulate real world scenarios regarding physics, interactions and their behavior and properties [12]. Seth et al. therefore expanded existing definitions of VR in regards to product design and described it as “the capability to assemble virtual representations of physical models through simulating realistic environment behavior and part interaction to reduce the need for physical assembly prototyping resulting in the ability to make more encompassing design/assembly decisions in an immersive computer-generated environment.”

Virtual reality applications are often used in the automotive sector to experience [17] or even drive the desired car in the virtual environment [21]. Nissan also enables consumers to even design their own Nissan car in VR [29]. Companies such as L’Oreal and Unilever [16] have created virtual reality store simulations to enable better shelf layouts and therefore increase sales [31]. Companies like IKEA [22] or Lowe’s [26] use VR applications to design and explore living rooms and the furniture within and even share it on YouTube to increase the social experience of the customer. In the tourism sector, travel agencies and hotels use virtual reality to market and manage destinations [30]. These so called “virtual tours” are often panoramic pictures or videos that do not allow free navigation. Theme parks are also an increasing market for VR entertainment. Various theme parks around the globe already offer attractions that are available via VR and AR technology [18].

These various examples show that VR has actually got very far in increasing the visual quality of its generated content. In contrast, they also show that the main focus in VR has been the visual stimuli and that the integration of other sensorial channels such as haptic feedback or smell is still in its infancy. Nevertheless, VR already gives many people an immersive experience with a measurable effect on their UX.

2.3 Prototyping

Prototypes of products are simplified representations of design ideas before the final artefacts are created. The most common usage of a prototype is as a “learning tool” and can be used at any stage of the design process to find issues and explore new ideas [13]. Especially for innovative, novel concepts, the creation of prototypes, i.e. prototyping, is an important prerequisite for the evaluation of user experiences. Already in the early phases of product development, assessments of the success potential of a product idea can be obtained. Insights into promising and less promising ideas as well as possible technical realizations help to reduce development risks and thus costs. In addition to their use in evaluation, prototypes also serve to demonstrate design decisions and possible design alternatives. Thus, prototyping offers great potential from a company's point of view and is a major activity in product development [15]. Prototypes do not need to be perfect. They range from low-fidelity drawings to high-fidelity models at several levels, all designed to communicate and explore the quality and the design of the product [13]. Prototypes modelled with virtual reality technology can be seen as experience prototyping [10]. It describes the simulation of a product experience through involvement of the consumer. As in other prototyping approaches, the focus is also on the question of how a product idea plays out on the experience level, whereby usability aspects and questions of interaction design can also be examined [15].

3 Considerations for the Study Design

In order to compare the perceived user experience of real products and virtual representations, a comparable test set-up was created in reality and in VR.

3.1 Virtual Environment

It was important for this test to achieve a natural and comfortable interaction for the user, especially if the participant has limited or no previous experience with VR technology. The virtual environment was designed to create a high level of immersion for the user. Therefore, the virtual room was built similar to the real office in which the participants were located while using the VR devices. The VR setup used in this experiment was the HTC VIVE Headset [39] and the associated controllers. The virtual environment was created with the Unity Game Engine [34]. After all objects and products had been modeled with Blender [6] and Fusion 360 [4], they were imported into Unity to create the virtual set-up.

3.2 Products

As already mentioned, the product test contained three different products that are independent from each other. The choice for the three products was based on the following parameters: The objects should be part of the same environment, the interactions should be just physical, the level of interaction should be different from low to higher complexity and the objects should not be already known by the participants. The selected objects where part of a new lab-inventory which was introduced months after this study:

Lamp

The first object is an interactive lamp created and designed at a university. It can be moved upwards and downwards by grabbing and dragging the lamp in the desired direction. The light can be activated as well as deactivated via the black motion sensor on one of the sides. In addition, the LED light bars can be rotated in 360°. It is the object with the highest grade of interactive complexity (Figs. 1 and 2).

Fig. 1
figure 1

Screenshot of the lamp in VR

Full size image
Fig. 2
figure 2

Photo of the lamp

Full size image

Convertible Table

The second product is a convertible table that can be used either as a standing table or as a normal desk. To change its purpose, the table just has to be turned over its wheel axis (Figs. 3 and 4).

Fig. 3
figure 3

Screenshot of the table in VR

Full size image
Fig. 4
figure 4

Photo of the table

Full size image

Miura Barstool

The third product is a barstool designed by Konstantin Grcic. It is the object with the lowest functionality and intractability (Figs. 5 and 6).

Fig. 5
figure 5

Screenshot of the barstool in VR

Full size image
Fig. 6
figure 6

Photo of the barstool

Full size image

Testing Room

For the testing we chose an office room inside the university. The choice was made because the three objects are used for agile work in office context. Additionally, the whole setup enabled a rational and undisturbed testing process with the experience in one room and the questionnaire taking place in the next one.

The main reason that the VR-twin of the room was designed as similar as possible to the real one was to have as few side effects on the VR-participants’ experience as possible when they used the VR-Headset; it should feel as if there was no change in their situation (Figs. 7 and 8).

Fig. 7
figure 7

Screenshot of the testing room in VR

Full size image
Fig. 8
figure 8

Photo of the testing room

Full size image

3.3 Questionnaire

The research methodology of this survey questionnaire has a mainly quantitative focus, but also contains some qualitative questions [1, 24]. The first page of the survey has been created to collect the necessary demographic data of the participants. The participants were asked to provide information about their gender, age, profession and annual income.

It is common practice to evaluate the user experience of interactive products via questionnaires. The User Experience Questionnaire (UEQ) [27] and the AttrakDiff 2 [20] are two of the most used tools to evaluate interactive products by questionnaires. Both constructions rely on the theoretical framework of user experience that distinguishes between pragmatic quality, hedonic quality and perceived attractiveness of a product. Furthermore, both questionnaires use a format of a seven-stage scale with two opposite adjectives on each end. A remarkable difference between both questionnaires is the selection of word pairs. The AttrakDiff 2 questionnaire has proven to be a strong and reliable source of measurement of perceived awareness of products as well as for user experience measurement with a strong background in research [2, 23, 32], and for that reason we selected it to evaluate pragmatic quality, hedonic quality and perceived attractiveness.

For each product, the AttrakDiff 2 questionnaire [20] was used to measure the perceived awareness of hedonic quality—identification, hedonic quality—stimulation, attractiveness as well as the perceived pragmatic quality. The questionnaire has proven to be a strong and reliable source of measurement of perceived awareness of products as well as for user experience measurement [2, 23, 32].

The questionnaire consists of twenty-eight 7-point adjective pairs. The word pairs represent strong contrasts and can be subdivided into further rating levels. In addition to the AttrakDiff 2, the participants were asked to rate on a 5-point scale how likely it was that they would buy the product for themselves, or recommend it to friends or colleagues. Furthermore, two qualitative questions were asked at the end of each product evaluation. The participants were tasked to indicate a fair price for each product and state what they liked and disliked about the objects.

For the virtual reality evaluation, two additional pages were added to the questionnaire to rate the VR-application, the devices and the general attitude of the participant towards VR and new technologies. Thus, the first page contained questions about the user acceptance of the virtual reality technology used in the product test. The Technology Acceptance Model [14] was used for this task and included questions about perceived ease of use [14, 36], perceived usefulness [25], attitude towards using the technology [37] and the behavioral intention to use it [37].

The second page therefore contained questions about the attitude towards new technologies, the intention to use and buy VR technology and the general usage of existing VR technology.

4 Method

The test was conducted as a participatory and summative evaluation in the rooms of the University of Applied Sciences Neu-Ulm. In VR we tested 33 (average age 23; 15 male, 18 female) and in the real environment we tested 31 (average age 24; 15 male, 16 female) students of the University of Applied Sciences Neu-Ulm. The target group was deliberately chosen homogeneously in order to exclude as many target group-specific disruptive factors as possible. All participants tested out the same three products and evaluated them afterwards. Pre-test instructions were similar for every participant in the VR test as well as the real product experience. Therefore, a guideline was created that contained all the important indications for the students. At the start of the experiment, everyone was told that they were participating in a product test and would be asked to evaluate three products afterwards via a questionnaire.

In order to learn about all the functions of each product, the participants were introduced to a scenario and had to carry out specific tasks for each object (four tasks for the lamp, four tasks for the table and two tasks for the barstool) that were described in the scenario.

In addition, the VR students were taught how to grab and move objects within the virtual environment (Fig. 9). In order to create the same prerequisite for every participant, an additional exercise was created to teach the user how to grab and control objects inside the virtual world. The users were tasked to sort colored balls into the according bowls. After achieving that, the instruction test was disabled and access to the three products was activated to the user.

Fig. 9
figure 9

Screenshot of the teaching exercise

Full size image

The users were instructed to take as much time as necessary for each product as they had to evaluate the usability as well as their impression of each product afterwards. It was also mentioned that they were completely safe within the experience and could not do anything wrong inside the room. After the users experienced all the products, they were asked if they thought that they have seen enough of every product to evaluate it, otherwise they could look at a specific product again. After every participant had mentioned that they had seen enough, the product test was finished and they were asked to evaluate their product experience. The participants were taken to another room to rate the three products via the associated questionnaire. They were tasked to answer the questions spontaneously without thinking too much about it.

5 Results

To verify the reliability of the AttrakDiff2 subscales, Cronbach’s Alpha was calculated for the four subscales for every product. Usually, an Alpha-Coefficient of >0.7 is considered a good indicator for the reliability of a scale. In our case, every resulting value of each product is above this threshold and therefore we can derive that the four subscales of the Attrakdiff2 questionnaire show a high reliability for our data.

5.1 Lamp

Figure 10 highlights the mean scores of the lamp for the four subscales pragmatic quality (PQ), hedonic quality—identification (HQI), hedonic quality—stimulation (HQS) and attractiveness (ATT). Analyzing the results by means of t-tests shows only slight and insignificant differences between the real and the VR outcomes. In the pragmatic quality, the real product is ranked slightly better than the VR counterpart (MReal = 4.6; SDReal = 0.85; MVR = 4.5, SDVR = 0.80; t = −0.259; p = 0.800). In the hedonic quality identification (HQI), the virtual reality object was perceived slightly better than the real product (MReal = 5.1; SDReal = 0.4; MVR = 5.3, SDVR = 0.7) t = 0.7, p = 0.5. This applies also to the hedonic quality stimulation (MReal = 5.5; SDReal = 0.5; MVR = 5.7, SDVR = 0.6) t = 0.4, p = 0.7, and also to the overall attractiveness of the products (MReal = 5.3; SDReal = 0.3; MVR = 5.4, SDVR = 0.3) t = 1.1, p = 0.3. Figure 9 shows the extended profile of word pairs with the mean values for both scenarios.

Fig. 10
figure 10

Mean values for the lamp (PQ—pragmatic quality; HQ-I and HQ-S—hedonic quality individual and stimulation; ATT—attractiveness)

Full size image

The balanced user experience of the lamp in both scenarios is also reflected in the intention to buy or recommend the product. 58% of the VR participants were likely to buy the product to a fair price, whereas 53% that participated in the test of the real product answered with the same outcome. In addition, 58% (VR) and 63% (real) were likely or extremely likely to recommend the product.

However, the individual feedback of the lamp pointed out differences in the awareness of the product. The positive aspects of the object were similar in both cases. The participants rated the lamp as modern, futuristic and innovative. They enjoyed the form and style of the object and liked the rotatable LED lights. The negative aspects varied in both scenarios though. Regarding the real-life object, attendees complained about the sound that appears when the object was adjusted in height. Furthermore, several students thought it was cumbersome that the sensor of the light was difficult to find when the lamp was in a higher position. On the other hand, participants of the VR object mostly pointed out that it could be sometimes difficult to grab the LED lights with the controller, especially when the object was moving/turning.

5.2 Convertible Table

Figure 12 highlights the mean scores of the convertible table for the four subscales. Similar to the lamp, an analysis of both outcomes shows no significant difference between both versions of the table. In the pragmatic quality, the real product is ranked slightly worse than the VR counterpart (MReal = 5.3; SDReal = 0.9; MVR = 5.4, SDVR = 0.9) t = 0.09, p = 0.9. In the hedonic quality identification, both objects are ranked equally (MReal = 4.4; SDReal = 0.4; MVR = 4.4, SDVR = 0.5) t = 0.05, p = 0.9. This applies also to the attractiveness (MReal = 4.4; SDReal = 0.5; MVR = 4.4, SDVR = 0.4) t = 0.0, p = 1. In the hedonic quality stimulation, the real object is ranked slightly better (MReal = 4.0; SDReal = 0.4; MVR = 3.8, SDVR = 0.3) t = −1.2, p = 0.2. Figure 11 shows the extended profile of word pairs with the mean values for both scenarios (Fig. 12).

Fig. 11
figure 11

Results of the Attrakdiff2 for the lamp (PQ—pragmatic quality; HQ-I and HQ-S—hedonic quality individual and stimulation; ATT—attractiveness)

Full size image
Fig. 12
figure 12

Mean values for the table (PQ—pragmatic quality; HQ-I and HQ-S—hedonic quality individual and stimulation; ATT—attractiveness)

Full size image

Similar to the lamp, the balanced awareness of the table is also reflected in the intention to buy or recommend the product in both scenarios. 15% of the VR participants are likely to buy the product for a fair price, whereas 23% that participated in the real-life product test answered with the same outcome. Furthermore, 30% (VR) and 33% (real) are likely or extremely likely to recommend the product.

The individual feedback this time, in contrast to the feedback about the lamp, points out that the participants perceived the product similarly in VR as well as in real life. Positive aspects about the product were that it is practical, simple and extremely space saving. On the other hand, the attendees mostly criticized the cold design of the object and mentioned that it was not aesthetic enough.

5.3 Miura Barstool

Figure 14 highlights the mean scores of the barstool for the four subscales. In contrast to the other two objects, the results of the barstool show significant differences between the two scenarios in the scales HQI and ATT. In the pragmatic quality, the VR product achieved a higher score than the real object (MReal = 4.7; SDReal = 0.5; MVR = 5.2, SDVR = 0.6; t = 1.6; p = 0.1). This applies also to the hedonic quality identification (MReal = 4.4; SDReal = 0.3; MVR = 5.1, SDVR = 0.4; t = 3.4; p = 0.005), and also to the hedonic quality stimulation (MReal = 4.3; SDReal = 0.4; MVR = 4.8, SDVR = 0.6; t = 1.8; p = 0.09), Finally, the VR product also achieved a much higher score in the attractiveness (MReal = 4.6; SDReal = 0.3; MVR = 5.2, SDVR = 0.2; t = 3.7; p = 0.003). Figure 13 shows the extended profile of word pairs with the mean values for the barstool (Figs. 14 and 15).

Fig. 13
figure 13

Results of the Attrakdiff2 for the table (PQ—pragmatic quality; HQ-I and HQ-S—hedonic quality individual and stimulation; ATT—attractiveness)

Full size image
Fig. 14
figure 14

Mean values for the barstool (PQ—pragmatic quality; HQ-I and HQ-S—hedonic quality individual and stimulation; ATT—attractiveness)

Full size image
Fig. 15
figure 15

Results of the Attrakdiff2 for the barstool (PQ—pragmatic quality; HQ-I and HQ-S—hedonic quality individual and stimulation; ATT—attractiveness)

Full size image

The VR favored awareness of the barstool is also reflected in the intention to recommend and buy the product. 51% of the VR participants were likely to buy the product at a fair price, whereas only 20% that participated in the real product test answered with the same outcome. In addition, 42% (VR) and 23% (real) were likely or extremely likely to recommend the product.

The individual feedback showed several differences between the two tests. The virtual object was described as an uncomplicated object with an attractive design. Almost no negative aspects were mentioned for the product in VR, only some students mentioned that it might look a bit uncomfortable but it was hard to assess. The perception of the real product was more focused on the comfort of sitting on the barstool instead of the design. While a few participants still mentioned that it looks quite aesthetic, many criticized the material and judged the shaky way the barstool behaved.

6 Conclusion

A first result of the study is that there is no significant difference in the evaluated experience of the lamp and the convertible table in virtual reality compared to the real-life objects. Both products were rated nearly the same (maximal difference 0.2) in the scales of pragmatic quality, hedonic quality identification, hedonic quality stimulation and attractiveness. Qualitative feedback of the table has, in fact, shown that the participants liked and criticized the same aspects of the product in both scenarios.

On the other hand, the evaluation of the lamp, which is a more complex and interactive product, demonstrated that the users valued the object in both representations nearly the same, but criticized different aspects in both cases. While the hover-function to enable the light was the only possibility in VR to activate it, it was not seen as a problem. However, participants of the real-world test found it cumbersome to activate it when the sensor was outside the range of vision.

A second finding is that the barstool was evaluated better in VR compared to the real counterpart in all four subscales (minimum difference 0.4; maximum difference 0.7). Furthermore, the results showed significant differences in the scales of attractiveness and hedonic quality identification. The qualitative individual feedback reflects the stated outcome. While most of the VR participants only pointed out the positive aspects of the barstool, students who tested out the real object criticized the material and described it as uncomfortable and shaky.

The positive results of the technology acceptance model can be an indicator that the VR technology and the associated product test in VR did not influence the perception of the products in a negative way. On the other hand, the virtual reality application did not seem to cause unusual side effects that would influence the students to rate products better in VR as there are no significant differences in two of the three products.

These results give a first glimpse of the possibilities VR can offer designers for virtualization of certain steps in a development process. It can be stated that the evaluation of these product representations in VR led to a comparable UX with the real products when they had a comparable set of features and interactions in both spaces, even though the fidelity of the virtual space differed.

7 Limitations and Further Research

In order to provide an immersive experience, one of the chosen limitations of the study was that participants experienced all three products at once and evaluated them afterwards. In this set-up participants have to remember which object felt which way. Even though the testing time for all products was short and the order was randomized, through the post hoc evaluation participants might have rated all products more similarly because the experience that they remembered was mixed.

It might also be argued whether the obtained results would have been different if the VR device were more immersive than the current setup. The current grabbing mechanics are via the VIVE controller. Newer and more immersive technology like VR haptic gloves could achieve a more natural and therefore more immersive way of interacting with the objects in virtual reality. Thus, it would be interesting to increase the fidelity of the VR application. The current set-up was built by one person with a limited amount of resources in VR. The whole experience could be even more immersive with a team of VR experts and the necessary budget.

Another factor that should be investigated in future studies is varying target groups. In both cases, only students participated in the product tests. The average age was 23 (VR) and 24 (real) which does not represent the majority of the general population. It might be argued that the results could have been completely different with an older average age. As a result of testing only students, the average annual income was in the bracket between 0€ and 5000€. This could affect the answer whether the participant would be likely to buy the product.

Even though the study could show certain design aspects that could be simulated with digital VR prototyping tools, an additional question would be what design aspects still need a real prototype to evaluate the UX, especially since the results of the barstool evaluation pointed towards a difference in perceived functional qualities for objects where some of their interactions cannot be simulated in VR.

The results of this study show the potential for product prototyping and evaluation in VR, but there is a need for further research to exactly identify the product parameters which lead to comparable UX-evaluations in VR.