Influence of Demographic Variables and Usage Behaviour on the Perceived User Experience

Abstract

Users form an overall impression concerning the user experience (UX) based on their perception of special UX qualities. Therefore, measuring users’ perceptions of these particular UX aspects is essential for determining the UX of a product. The measured hedonic qualities, e.g. stimulation or aesthetics, and pragmatic qualities, e.g. efficiency or learnability, form a suitable overall impression of the perceived user experience of the product. In practice, the measurement of such qualities is often carried out with the help of standardized questionnaires such as the SUS, UMUX, or UEQ. However, the same product sometimes shows large differences in the ratings of different users. It is conceivable that other factors, for example, demographics, usage frequency, or experience with a product, can influence UX ratings. In a previous study (Kollmorgen, Schrepp & Thomaschewski, 2022), the four products Netflix, Microsoft PowerPoint, BigBlueButton, and Zoom were examined for differences in the UX ratings according to such factors. In the present paper, the data set was extended by two additional products of different product categories to deepen and broaden the investigation of the influences of external factors on the perceived UX of products, with a specific focus on their impact on pragmatic and hedonic qualities.

1 Introduction

To evaluate how well products meet the requirements of their users, questionnaires are often used. Standard questionnaires like the User Experience Questionnaire (UEQ) (Laugwitz, Schrepp & Held, 2008), the Usability Metric for User Experience (UMUX) (Finstad, 2010), or the System Usability Scale (SUS) (Brooke, 1996) can be used to measure the usability and user experience (UX) of products. This makes it possible to align the needs of users as closely as possible with the products (Schrepp, 2021).

To gain an appropriate overall impression of the measured products, it is important to distinguish between hedonic and pragmatic factors (Hassenzahl, Diefenbach & Göritz, 2010). Pragmatic qualities (PQ) are associated with a product's ability to assist users in achieving specific goals, while hedonic qualities (HQ) are geared towards fulfilling psychological needs that go beyond the sole purpose of task completion, such as stimulation or aesthetics (Hassenzahl, 2008; Winter et al., 2017).

However, results often show that different users do not perceive the user experience or usability of the same product in the same way. This could be attributed to several factors. On the one hand, studies have already shown that the importance of hedonic and pragmatic UX factors depends on the product category (Winter et al., 2017; Kollmorgen et al., 2021; Meiners et al., 2021; Schrepp et al. 2023). In one study, for example, it became clear that for the product category of online banking, pragmatic UX factors such as trust or quality of content were rated as important, in contrast to hedonic factors, such as stimulation or aesthetics. On the other hand, also a different usage behaviour can have an impact on the perceived usability and user experience of a product. E.g., people who use a product more frequently typically know it better, have adjusted their usage behaviour to avoid typical UX problems of the product, and therefore perceive the user experience differently. Conversely, a product is presumably only used more frequently if it offers a good user experience.

This led to the first research question, RQ 1: Are there external factors besides the classic UX factors that influence the perceived user experience of a product and assist in explaining the differences in UX ratings?

However, the way the product is used can have varying effects on pragmatic and hedonic factors. While having a high level of expertise with a product is likely to lead to higher ratings for pragmatic quality, it is uncertain if this same effect applies to hedonic qualities.

Based on this research question, the study by Kollmorgen, Schrepp and Thomaschewski (2022a) investigated which impacts external factors can have on the pragmatic and hedonic qualities of well-known products. For this purpose, four products from three different product categories were selected, which have been heavily used in recent years. The streaming platform Netflix, the video conferencing tools Zoom and BigBlueButton, and the presentation software Microsoft PowerPoint. These products support leisure activities at home as well as remote working and thus display a quite heterogenous set of use cases and user experience factors.

Building on that, this paper extends and deepens the findings of this first study from Kollmorgen, Schrepp and Thomaschewski (2022a) by collecting data on two other products that are also heavily used and well-known: the social network platform TikTok and the online banking software PayPal. TikTok, just like Netflix, is mainly used for leisure and thus should have a stronger focus on hedonic qualities such as fun and visual aesthetics. PayPal, on the other hand, has more pragmatic purposes and focuses mainly on the efficient fulfilment of working tasks. This product selection ensures that the influences on both hedonic and pragmatic UX factors are reviewed and deepened with two additional product categories, resulting in six products of five product categories overall.

This led to the overarching second research question, RQ2: To what extent are the pragmatic as well as the hedonic quality of products influenced by the external factors mentioned above? Does the impact of these factors influence pragmatic and hedonic qualities differently?

This paper is structured as follows: After a presentation of the UX questionnaires used in Sect. 2, the interindividual differences in the perception of UX are explained in Sect. 3, which serve as the basis for answering the research questions. The methodology of the two studies developed on this basis is then explained in Sect. 4. The results of these are presented in Sect. 5 and form the basis for answering the two research questions in Sect. 6, concluding the article in Sect. 7 with a summary and outlook.

2 UX Questionnaires

The goal of this research is to investigate the influence of demographic factors and differences in the usage experience or usage frequency on the subjective impression of persons concerning UX. The standard method to measure such subjective UX impressions are questionnaires. But user experience itself is a quite heterogeneous concept that contains many facets. There are many established standard UX questionnaires (see Schrepp, 2021a for an overview) available that measure aspects of UX, but they deviate to some extent from the specific UX aspects they consider. For that reason, the studies use three quite common UX standard questionnaires that will be shortly introduced in this section.

2.1 System Usability Scale (SUS)

The SUS (Brooke, 1996, 2013) is a short questionnaire that focuses on the measurement of classical usability aspects, for example, usefulness, consistency, or ease of learning. The original publication announced the SUS as a “quick and dirty usability scale”. But despite this modest description, the SUS is currently still one of the most used usability questionnaires and there is a huge number of papers that investigate the psychometric properties of the SUS (see Lewis, 2018 for an overview).

The 10 items of the SUS are short statements that describe aspects of usability:

• 1. I think that I would like to use this system frequently.

• 2. I found the system unnecessarily complex.

• 3. I thought that the system was easy to use.

• 4. I think that I would need the support of a technical person to be able to use this system.

• 5. I found the various functions in this system were well integrated.

• 6. I thought there was too much inconsistency in this system.

• 7. I would imagine that most people would learn to use this system very quickly.

• 8. I found the system very cumbersome to use.

• 9. I felt confident using the system.

• 10. I needed to learn a lot of things before I could get going with this system.

The SUS contains a single scale and produces an overall score between 0 and 100. Each item can be rated on a 5-point agreement scale with the endpoints Strongly Disagree (left) and Strongly Agree (right).

For items 1, 3, 5, 7, and 9 agreement represents a positive evaluation, and these items are scored as 0 to 4 from left to right. For items 2, 4, 6, 8, and 10 agreement represents a negative evaluation, and these items are scored as 0 to 4 from right to left, thus in the opposite direction. Thus, a 4 represents the most positive evaluation, and a 0 the most negative evaluation. The scores for the 10 questions are added up to a participant score between 0 and 40, which is then multiplied by 2.5 to scale it between 0 and 100 (the argumentation for this rescaling is that a score between 0 and 100 is easier to communicate). The SUS score for a product is then simply the average over all participant scores.

2.2 Short Form of Usability Metric for User Experience (UMUX-LITE)

The UMUX-LITE (Finstad, 2010) is a short usability questionnaire that contains the two items:

• This system’s capabilities meet my requirements.

• This system is easy to use.

The measurement concept of the UMUX-LITE is related to the Technology Acceptance Model (Davis, 1986). This concept assumes that user acceptance of a new technology is based on its perceived usefulness (first item of the UMUX-LITE) and perceived ease of use (second item of the UMUX-LITE).

Participants can rate these items on a 7-point response scale with the endpoints Strongly disagree (left) and Strongly agree (right). The responses are scored as 0 to 6 from disagreement to agreement, therefore 0 is the most negative, and 6 the most positive evaluation. Just like in the SUS, the item scores are added up to a participant score between 0 and 12. This score is then rescaled to 0 to 100 by dividing it by 12 and multiplying it by 100. The UMUX-LITE score for a product is then the average over all participant scores.

The UMUX-LITE thus provides a high-level measurement of UX related to the concept underlying the technology acceptance model.

2.3 Short Form of the User Experience Questionnaire (UEQ-S)

The original User Experience Questionnaire (UEQ) (Laugwitz, Schrepp & Held, 2008) measures UX by six pragmatic and hedonic UX aspects (Attractiveness, Efficiency, Perspicuity, Dependability, Stimulation, Novelty). The UEQ contains 26 items in the form of a semantic differential.

A short form with just 8 items was developed (Schrepp, Hinderks & Thomaschewski, 2017) to support use cases that require short completion times. This short version (UEQ-S) contains only two scales for pragmatic (task-related UX qualities) and hedonic (non-task-related UX qualities).

The items of the UEQ-S are:

The first 4 items form the scale for pragmatic quality and the second four items the scale for hedonic quality. An overall value is determined by the mean over all 8 items, it represents to overall impression concerning UX. The items are scored from −3 (negative term) to + 3 (positive term). The scale scores are simply the mean over all items in the corresponding scale and all participants in a study.

The UEQ-S questionnaire and all supporting material (handbook, translations in more than 30 languages, and an Excel-based data analysis tool) are available free of charge at https://www.ueq-online.org/.

2.4 Differences Between the Three Questionnaires

When analyzing the items of SUS, UMUX-LITE, and UEQ-S semantically, it is clear that all three questionnaires measure distinct concepts of UX. The SUS measures only classical usability criteria, for example, ease of learning, consistency, efficiency, or controllability, thus setting the focus on aspects that support or hinder users to work on their tasks. This aspect is considered also in the UMUX-LITE (measured here in a single question This system is easy to use) and in the UEQ-S (in the four items of the scale for pragmatic quality). The UMUX-LITE covers in addition to usability the usefulness of a product. The UEQ-S covers in addition to pragmatic quality also the hedonic quality or fun of the use of a product (by the four items of the scale hedonic quality). For the research questions of this study, this is a relevant aspect, as demographic factors or usage experience and frequency may only impact specific UX aspects and not all of them.

3 Interindividual Differences in the Perception of UX

Of course, different persons have different perceptions of the UX of a product. A nice example to demonstrate this can be found in Rummel & Schrepp (2018). Figure 1 (presentation taken from Schrepp, 2021b based on data described in Rummel & Schrepp, 2018) shows the distribution of ratings, grouped into intervals of length 10, obtained with the System Usability Scale (SUS) for three products.

Fig. 1.

Relative frequencies of individual SUS scores for three different example products X, Y, and Z (Rummel & Schrepp, 2018 and Schrepp, 2021b).

Product Z is clearly rated much better than Product X. But it is interesting to note that the observed ratings for both products span the entire range. Thus, even for the on average poorly rated Product X, there are users that give quite high ratings. And for the on average good-rated Product, Z there are some strongly dissatisfied users. What are the reasons for such strong interindividual differences in the perception of UX?

Of course, demographic factors, for example, gender, age, or the cultural background of a user, can influence the UX perception of a product. But a majority of studies concerning the SUS (see Lewis, 2018 for a summary) found no effect of age and gender on SUS ratings. For the UEQ (Laugwitz et al., 2008), a recent study (Aufderhaar et al., 2019) found no substantial differences between the ratings of men and women for some websites.

Personality traits, usually conceptualized based on the Five-Factor Model of personality (John & Srivastava, 1999; McCrae & John, 1992), can also influence ratings of UX questionnaires (for example Kortum & Oswald, 2018; Liapis et al., 2019; Devaraj et al., 2008; or Braun et al., 2019).

If users rate the UX of a product, they have to recall usage episodes from past interactions. Assume, for example, that a UX questionnaire asks about the speed of a system’s response to user inputs or commands. If users do not remember any long waiting times, they will rate this aspect as positive. If users remember a lot of situations where the system responds too slowly, the rating will be negative. Thus, the interaction history of a user with a product will of course have an impact on the perceived UX of that product.

The level of experience or the frequency of use may also impact the UX perception. A study by McLellan et al. (2012) found, for example, that experienced users tend to provide more positive UX ratings.

It is important to note that the impact of variables like age, gender, experience, or usage frequency on UX scores of standardized questionnaires depends on the concrete product. For general rules like, for example, “Gender has for all possible products no impact on S” (where S is a scale from a UX questionnaire) it will always be possible to find counterexamples. Assuming, for example, a website that is highly optimized for a purely female target group: will gender have an impact on UX ratings? If the design target is reached, then the answer will most likely be “Yes”. Therefore, products that are used for different usage scenarios are investigated in the studies.

4 Methodology

As already explained, the perceived user experience of a product can, on the one hand, depend on demographic factors or usage behaviour. On the other hand, these external factors can exert varying degrees of influence on pragmatic and hedonic qualities. For this reason, a total of six popular products from five product categories were considered (see Table 1). The assignment of products to specific product categories as well as their analogies in the importance of UX factors are formed below based on Meiners et al. (2021).

Table 1. Examined products with UX focus and product category.

4.1 First Study: Netflix, PPT, BBB and Zoom

The first study was published at the 19^th International Conference on Web Information Systems and Technologies (WEBIST) 2022 by Kollmorgen, Schrepp & Thomaschewski (2022a, 2022b). In this data collection, participants were recruited from various universities and through a panel and were compensated monetarily for their participation in the study. The target groups were provided with either German or English questionnaires between September and December 2021.

An online survey was conducted to gather data on the external factors influencing the four products. The survey begins with a brief set of instructions, followed by the collection of demographic information and usage behaviour details from the participants. Specifically, the following information was requested:

• Age

• Gender: Male (M), Female (F), Divers (D)

• Usage frequency (How often do you use < product name > ?): Not very frequent, Several times a month, Several times a week, On a daily basis

• Knowledge (How good is your knowledge of < product name > ?): Low, Medium, Strong, Excellent

• Duration of use (How long have you been using < product name > ?): Less than a week, Since more than a week, Since more than 6 months, Since more than a year, Since more than 5 years

Participants were not required to answer all the questions in the survey, which is why an additional “No answer” category was included. Following the section with demographic and behavioural questions, the survey included the two items from the UMUX-LITE, eight items from the UEQ-S, and ten items from the SUS as described above. At the end of the survey, participants were given the opportunity to provide free-form comments on the strengths and weaknesses of the product.

Completing this overall questionnaire, consisting of questions on demographic and usage data as well as on the items of the three questionnaires, took the respondents on average about 3 to 4 min. This shows that by using the short versions of the questionnaires, respondents were able to answer all questions and still spend very little time.

In this first study, the products Netflix, Microsoft PowerPoint (PPT), BigBlueButton (BBB) and Zoom were surveyed on demographics, usage patterns, and their perceived user experience.

We expected that the four products cover the range from pragmatic to hedonic quality. Netflix, belonging to the video streaming product category, is primarily used for private purposes and thus is expected to focus more on hedonic quality. In such cases of products that are primarily intended for private use, such as Netflix, hedonic factors like fun or beauty should not be neglected (Hassenzahl, 2001). On the other hand, PPT, from the product category presentation software, is used to complete work tasks, thus it is expected to have a strong focus on pragmatic quality. BBB and Zoom, both belonging to the video conferencing product category, are used for both private and professional purposes, which is why they should take into account both hedonic and pragmatic needs.

The obtained data sets were cleaned to enhance their quality. Any data records with a processing time that was too brief or too few clicks, or that had an incorrect response to the quality assurance question, were removed, resulting in the elimination of 97 records and leaving 338 records in total. For the four online surveys, the following numbers of responses were collected: Netflix (N = 97), BBB (N = 76), Zoom (N = 76) and Microsoft PowerPoint (N = 89). The participants had an average age of roughly 28 years, and more detailed information is available in the research protocol (Kollmorgen, Schrepp & Thomaschewski, 2022b).

4.2 Second Study: TikTok and PayPal

Building on the results of the initial data collection, the potential for deepening the answers to research questions was identified. There should be a stronger focus on assessing influences on hedonic and pragmatic quality. BBB and Zoom from the first study are both expected to focus equally on both pragmatic and hedonic quality, while Netflix (HQ focus) and PPT (PQ focus) only should depict one of the two. For this reason, the products TikTok (HQ focus) and PayPal (PQ focus) were selected as additions for the second data collection, as they both are expected to focus mainly on one quality each. TikTok, from the product category social network, is expected to focus more on hedonic quality due to the nature of the product, since, for example, UX factors such as aesthetics and novelty are considered important for this product category. PayPal, the online banking product category, is expected to place a correspondingly stronger focus on pragmatic quality, therefore pragmatic UX factors such as dependability and efficiency are more important here (Kollmorgen, Meiners, Schrepp & Thomaschewski, 2021).

Thus, in the first study, there were already two products that should equally focus on both pragmatic and hedonic quality, but only one product each that should focus more on PQ and HQ, respectively. The aim of adding these two products was therefore to deepen the statements on the influence on hedonic and pragmatic quality, in particular, since the entire range is then evenly covered.

The second study is therefore a replication study based on the first study from 2022. English questionnaires with the same structure were used, in which only the product names were changed to TikTok and PayPal. The surveys were again conducted via the panel and the respondents received monetary compensation.

After the corresponding data collection, the data were cleaned analogously to the first study. 27 records that did not meet the criteria were removed, resulting in a remaining set of 114 records for TikTok and 111 for PayPal. Detailed results can be found in the protocol (Kollmorgen, Schrepp & Thomaschewski, 2023).

5 Results

In the following, the data from both studies are considered and compared together in order to obtain more meaningful results. On the one hand, this is possible because it is a replication study, which means that the data from both surveys can be interpreted in the same way. On the other hand, sufficient data is also available for both new products so that analyses and interpretations can be carried out.

To ensure a meaningful interpretation of the influence of the demographic factors and usage behaviour on UX metrics, a lower limit for the number of participants in a category had to be defined. For example, it would not be meaningful to say that users who have only been using a product for a short time rate a product significantly good/bad if only 5 out of 100 respondents placed themselves in this category of usage frequency. As a lower limit, it was determined that a category under consideration must have at least N = 10 records. This is based on the fact that an average of 94 data records were collected per survey and a quantity threshold of 10% was set, which is established in statistical research. If the lower limit was not exceeded, the results were not interpreted and are shown in italics in the corresponding tables and diagonally patterned in the bars of the corresponding figures.

In the following, the results of the ratings are presented first, followed by the analyses of the demographic factor of gender as well as by the external factors usage frequency, knowledge, and duration of use. The results serve as the basis for answering the first research question RQ 1: Are there external factors besides the classic UX factors that influence the perceived user experience of a product and assist in explaining the differences in UX ratings?

5.1 Rating of the Products

First, the ratings of the products, in general, are to be discussed. For this purpose, the overall UX ratings of the individual questionnaires must be compared. To facilitate the comparison with the SUS and UMUX-LITE scales (from 0 to 100), the UEQ-S ratings (from −3 to + 3) were converted into percentages. This involved using a simple percentage calculation by scaling the values to 0–6, then multiplying by 100 and dividing by 6. The corresponding scaled scores are shown in Table 2.

Table 2. Scale values. Range 0–100. The UEQ-S scores were converted for better comparability. The UEQ-S measures the PQ and HQ in their own scales, which are shown separately here in the two lower lines.

Pragmatic Quality

Studies have shown with the help of high correlations that SUS, UMUX-LITE, and UEQ-S PQ Scale all measure a similar concept (Schrepp, Kollmorgen & Thomaschewski, 2023). This is also visible in Table 2 since the ratings show only minor differences. These results become clear in a summary ordering of the product ratings between the three questionnaires. In descending order were evaluated (see Table 2):

• UMUX-LITE: PayPal, Netflix, Zoom, TikTok, PPT, BBB

• SUS: Netflix, PayPal, Zoom, TikTok, PPT, BBB

• UEQ-S PQ Scale: PayPal, Zoom, Netflix, TikTok, BBB, PPT

That is why a comparison of the products can be made here first.

It is visible from Table 2 that PayPal is rated highest in terms of pragmatic quality for UMUX-LITE and UEQ-S. This may be related to the product's very strong focus on pragmatic quality in the online banking category (see also Meiners et al., 2021).

PPT and BBB, on the other hand, are rated the worst. It appears that Microsoft PowerPoint is perceived as too complex to effectively achieve goals. This is indicated by 21 out of the 37 open responses to the survey on PPT, which noted that the software's many different functions are overly extensive, complicated, or illogical. For instance, creating customized slide designs was mentioned as a particularly challenging aspect of using the software. Concerning BigBlueButton, it is often specified for use in the work/education environment, which reinforces the pragmatic focus, making users more critical in this regard. As a result, 8 out of 19 open responses to the BBB survey cited the absence of certain functions, such as the ability to control user volume, as the reason for their dissatisfaction.

Hedonic Quality

However, if we look at the UEQ-S HQ Scale, we get a different picture:

• UEQ-S PQ Scale: TikTok, Netflix, PayPal, Zoom, BBB, PPT

Looking at the hedonic quality (UEQ-S HQ Scale, Table 2), it is clear that TikTok is rated by far the best, followed by Netflix. As explained, both products are expected to have a hedonic focus, since they are used voluntarily in leisure time and are rarely prescribed by other people such as employers.

This is also a common outcome observed when evaluating HQ. The UEQ-S assesses the level of enjoyment and novelty associated with a product. However, since the products except for TikTok examined in the study have been available on the market for some time, they are considered less novel. As a result, design revisions are frequently employed in practice. This is one of the reasons why the HQ scores are notably lower in direct comparison to the PQ scores. This trend is also discussed in Sect. 6.

Even if one must not overinterpret these results, the influence of hedonic quality is evident in the corresponding products.

5.2 Impact of Gender

The initial analysis examines whether gender influences the ratings of SUS, UMUX-LITE, or UEQ-S scales for the six products. However, it is worth noting that there were overall only six self-identified diverse participants and three respondents who chose the “No answer” option, resulting in insufficient data to produce meaningful results for these categories. As a result, the focus will be on comparing the ratings between male and female participants. Table 3 displays the percentage proportions of male and female participants for all six product evaluations.

Table 3. Distribution of male and female participants.

Table 4 presents the values of the three UX questionnaires categorized by gender. Regarding the UEQ-S, the overall value is being used, which means the pragmatic and hedonic qualities are not considered separately at the moment.

The variation in ratings between Zoom and BBB is intriguing. Even though both products are in the same category and cater to similar use cases, there is a significant difference in the way females and males rate them across all three UX scales. Females rate Zoom much higher than males (see Table 4), whereas no such trend can be observed for BBB. It is possible that this is because BBB is predominantly used in an educational setting, whereas Zoom is a more versatile video conferencing tool that is employed for both personal and professional communication.

Table 4. Impact of gender on the 3 UX scales. Range 0–100 for UMUX-LITE and SUS, from −3 to + 3 for UEQ-S.

The gender of the participants had a statistically significant influence (ANOVA, p < .05) for Zoom on all three questionnaires. Female participants tended to rate Zoom better than male participants. For the other five investigated products Netflix, PPT, BBB, PayPal, and TikTok an analysis of variance (ANOVA, p < .05) showed that there is no statistically significant influence of gender on the scores. Detailed results can be found in the protocol (Kollmorgen, Schrepp & Thomaschewski, 2023).

Figure 2 illustrates the UEQ-S scores (from Table 4) categorized by gender. As can be seen, there are only small differences between the gender ratings. This difference is only significant for Zoom, but there is a slight tendency that female participants give higher ratings, except for Netflix (this is true for all three questionnaires). Therefore, due to the medium sample sizes, it cannot be ruled out that there is no effect of gender on the ratings, but in each case, the effect is quite small.

Fig. 2.

Influence of gender on the UEQ-S scores. Range from −3 to + 3.

5.3 Impact of Usage Frequency

As already explained, the perception and evaluation of UX can be influenced by the frequency of usage. When users actively engage with the product being evaluated more often, they are more likely to identify its features, advantages, and disadvantages. In addition, users may also adapt their behaviour to avoid known usability issues, which could be overlooked during their product evaluation, so that frequent users may rate the product better than non-frequent users.

Table 5 displays the percentage distribution of usage frequency across products, where the percentage distribution for Zoom, for instance, is already established by its product type: it targets students who have predetermined times during the course of modules in which they use the product.

Table 5. Distribution of usage frequency.

Table 6 further examines usage frequency and displays the values for the three questionnaires. Usage frequencies with fewer than 10 participants are shown in italics.

As observed, the more frequently a product in these categories is used, the better the UX score in the questionnaires is. This correlation is not surprising, as good UX tends to result in increased usage frequency, and over time, users with more frequent product usage are likely to have a better impression.

An ANOVA (p < .05) showed that the frequency of usage had a significant impact on the SUS scores for Netflix, Zoom, and TikTok. In addition, a significant impact on the UMUX-LITE scores for Zoom, BBB, and TikTok as well as for the UEQ-S scores for Netflix and TikTok could be found.

Table 6. Impact of usage frequency. Range 0–100 for UMUX-LITE, and SUS; from −3 to + 3 for UEQ-S. Usage frequencies with fewer than N = 10 participants are shown in italics.

Figure 3 depicts the SUS scores (from Table 6) for the six products investigated, in relation to self-reported usage frequency. Usage frequencies with fewer than 10 participants are shown diagonally patterned. Many of the differences in scores are relatively high i.e., the impact on usage frequency on the scale scores also leads to meaningful differences. It is noteworthy that Netflix, Zoom, and PayPal are consistently rated higher than PPT, BBB, and TikTok in all usage frequency categories.

Fig. 3.

Influence of usage frequency on the SUS scores. Range 0–100. Usage frequencies with fewer than 10 participants are shown diagonally patterned.

5.4 Impact of Knowledge

It is also possible that experience with the range of products being evaluated could impact the evaluation process. Similar to increased usage frequency, greater knowledge of the products may lead to a clearer identification of their advantages and disadvantages.

Table 7 displays the percentage distribution of self-reported knowledge among the participants.

Table 7. Distribution of self-reported product knowledge.

Consistent with previous observations, it became evident that Netflix, Zoom, and PayPal receive better ratings overall. However, TikTok clearly stands out in terms of this external factor, scoring the best overall, except for users who have little knowledge of the product. PPT and BBB again achieve the worst UX ratings.

Table 8 displays the UX ratings for the three questionnaires based on the reported knowledge of the products, which was the basis for the calculations. Knowledge with fewer than 10 participants is shown in italics.

Table 8. Distribution of self-reported product knowledge. Range 0–100 for UMUX-LITE, and SUS; from −3 to + 3 for UEQ-S. Knowledge with fewer than N = 10 participants is shown in italics.

Additionally, the assumption that greater experience with the products can lead to better evaluations is further supported. On average, participants rated the products more positively when they reported greater knowledge of them. This trend is clearly visible across all three questionnaires. Figure 4 also displays a graphical representation of this trend for UEQS (from Table 8). Usage frequencies with fewer than 10 participants are shown diagonally patterned.

Fig. 4.

Influence of knowledge on the UEQ-S scores. Range from −3 to + 3. Knowledge with fewer than 10 participants is shown diagonally patterned.

With the help of various ANOVA analyses (p < .05), statistically significant influences were also found for the external factor knowledge. For the UMUX-LITE there is, except for Netflix and PayPal, a significant impact of knowledge on the scores. For SUS the impact is significant, except for PPT and PayPal. For the UEQ-S there is only a significant impact of knowledge observed for TikTok.

5.5 Impact of Duration of Use

It is reasonable to assume that users who have been using a product in these categories for an extended period may have a better understanding of it. This does not necessarily imply that they know all of the product's functions and can operate it flawlessly, but rather that they can navigate it based on their needs. Conversely, users who have only used a product for a short time may struggle to achieve their goals. It is necessary to investigate the impact of usage duration on the ratings.

For the duration of use (see Table 9) there is for most products one category that clearly dominates. Thus, it is not really a surprise that there is for most products no significant impact of this variable on the scores. An exception is TikTok, where the duration of use indeed significantly influenced the questionnaire scores according to an ANOVA (p < .05).

The distribution of responses in terms of duration of use is shown in Table 9. It is visible that most categories contain fewer than 10 participants. Therefore, no further statements about the data are made. The complete data can be found in the protocol (Kollmorgen, Schrepp & Thomaschewski, 2023).

Table 9. Distribution of duration of use.

6 Discussion

In the following, the results are discussed in order to answer the two research questions.

• RQ1: Are there external factors besides the classic UX factors that influence the perceived user experience of a product and assist in explaining the differences in UX ratings?

• RQ 2: To what extent are the pragmatic as well as the hedonic quality of products influenced by the external factors mentioned above?

6.1 External Influencing Factors

To answer the first research question, the results of the two studies must first be considered in terms of the influence of external factors. Starting with gender, the results did not show a significant impact on the UX scale scores except for Zoom. Nevertheless, women tended to rate the products better in all surveys.

In contrast, significant influences of the usage frequency on the perceived UX could be demonstrated. Especially for the products Netflix, Zoom, and TikTok, influences by the usage frequency were found in different ways with the three questionnaires. This shows the affirmation that the more often a product is used, the better the perceived user experience is, and vice versa.

Influences were also shown regarding experience with the product (knowledge). Here, however, differences were more pronounced. Thus, all three questionnaires for TikTok found significant influence by the knowledge on the UX ratings. This may be related to the fact that TikTok is the youngest of the six products surveyed and focuses on innovation, which means that new functions are regularly provided by the social media platform. Accordingly, with a better experience with TikTok, more benefits can be perceived in the UX. For PayPal, in contrast, no significant influence of knowledge was shown in the scores of the questionnaires. This may be related to the nature of the product. As an online payment service, PayPal only offers limited and intuitive functionalities. Thus, there is not much to learn, and an increasing experience may not cause better usability impressions. No clear trends emerged for the other products. For Zoom and BBB, significant influences by knowledge were found on ratings in the usability-focused questionnaires SUS and UMUX-LITE. For Netflix, significant results were found only in the SUS, and for PPT only in the UMUX-LITE. This may be due to the sample size as well as the selected target group (students).

No meaningful results could be obtained for the duration of use in this study. This is due to the varying extent of data records per duration category. The distribution of responses to the duration of use question corresponds to the maturity of the product. Microsoft PowerPoint and PayPal have been on the market the longest, which is why they show more data sets in the “More than 5 years” category. In contrast, BBB and Zoom have only gained popularity in the last few years, mainly due to the COVID-19 pandemic and the shift of work as well as social life to digital. The social network TikTok was also released only a few years ago and the study, therefore, shows only a few respondents who have been using the product since almost the beginning.

As anticipated, however, our study's findings show that usage frequency and knowledge are likely to have an impact on ratings, but that this impact depends on the concrete product. Although the majority of cases did not show significant effects for the other two factors, this could be attributed to the limited sample size and uneven distribution of participants in different categories. However, there was a discernible trend in the data.

In answer to the first research question, it can therefore be stated that there are definitely external factors that influence the perceived UX of products. Usage frequency could be proven as such an influencing factor, and trends were also visible for the experience with the product. Through these findings, another explanation for the differences in UX ratings of the same products can be confirmed.

6.2 Influences on Pragmatic and Hedonic Qualities

Finally, the identified significant influences of the external factors with respect to both the pragmatic and hedonic quality of the products are considered in particular to answer the second research question RQ2: To what extent are the pragmatic as well as the hedonic quality of products influenced by the external factors mentioned above?

Section 5.3 presented evidence that usage frequency significantly impacts product ratings, which is once again summarized in Fig. 5 (N < 10 patterned). Consequently, this external factor is examined once more in the context of pragmatic and hedonic quality, which is the focus of Fig. 5 (according to Table 6). The figure reveals a trend that is consistently visible with the overall UEQ-S ratings, particularly for pragmatic quality. This trend was also significantly observed concerning the PQ for Netflix, Zoom, and TikTok and the HQ for Netflix, BBB, and TikTok in the ANOVA tests (p < .05). The detailed information can be found in the Research Protocol (Kollmorgen, Schrepp & Thomaschewski, 2023).

However, some discussions of specific impacts on the pragmatic and hedonic quality can also be made with respect to the external factor knowledge. Significant impacts were found for the hedonic-focused product TikTok. The reasons for this are, as explained, that TikTok, as a young product, relies on users gradually becoming familiar with the innovative methods. If looked more closely at the results of the ANOVA, this is also reflected in the PQ/HQ scales of the UEQ-S. For TikTok, significant influences were found for both the PQ and the HQ. This speaks for the innovativeness of the product. As users become more familiar with TikTok, they learn the functions they need to fulfil (recreationally designed) their goals. At the same time, using the social network appeals to them more when they can better understand the functions. On the other hand, the novel functions of the platform seem overloaded for inexperienced users. Thus, people gave the worst ratings of perceived UX for TikTok when they rated their knowledge as poor.

For the products BBB and Zoom, which both cover hedonic needs in addition to pragmatic needs similar to TikTok, significant influences were found in two of three questionnaires. Both had a significant influence of the knowledge on the PQ, which can be justified by the prescriptiveness of the use in the professional environment and thus the more pragmatic focus of the persons surveyed.

For the pragmatically focused PayPal, on the other hand, no influences were found. As explained, one reason for this could be that PayPal cannot fulfil all the criteria for an online banking tool. For the pragmatically focused product PowerPoint, too, a significant influence was only found in one of the three questionnaires.

Overall, in answer to the second research question, this chapter demonstrated the extent to which the external factors can influence the pragmatic and hedonic quality of the products. It became clear that the external factors only influence or can influence specific UX aspects, but not all of them.

Fig. 5.

Influence of usage frequency on the UEQ-S overall, UEQ-S PQ and UEQ-S HQ scores. Usage frequencies with fewer than 10 participants are shown diagonally patterned.

7 Summary and Future Work

In this paper, it is argued that there are differences in the UX ratings of certain products that cannot be explained solely by their membership in different product categories. The resulting research question (RQ 1: Are there external factors besides the classic UX factors that influence the perceived user experience of a product and assist in explaining the differences in UX ratings?) was answered by conducting studies on a total of six products of different product categories using three short questionnaires. In these, questions on demographic and usage behavioural factors were asked to be able to examine correlations with the aid of analyses. The products examined in the studies, Netflix (HQ focus), TikTok (HQ focus), BigBlueButton (PQ/HQ focus), Zoom (PQ/HQ focus), Microsoft PowerPoint (PQ focus), and PayPal (PQ focus), cover different usage scenarios (see Table 1) and the importance of pragmatic and hedonic qualities, therefore, differs among the products. This creates the possibility to determine the influence of external factors on both scales in particular. This formed the basis for answering the second research question (RQ2: To what extent are the pragmatic as well as the hedonic quality of products influenced by the external factors mentioned above?).

With regard to the first research question, significant influences on the UX ratings could be determined for the usage behavioural factors usage frequency and knowledge. No conclusions could be drawn for the duration of use, as the distribution of responses was too heavily skewed towards one category in each case. In relation to the demographic factor gender, only an influence for the product Zoom was detected, so that at least a trend and possible influence of gender on specific products is visible. This represents a possible starting point for further correlation studies.

The external factors were then assessed to answer the second research question on the influence of pragmatic and hedonic quality. It was found that hedonic-focused products (TikTok, Netflix, BBB/Zoom) were rated better on average when users had a higher usage frequency. This is understandable since the products are used for leisure and thus are not prescribed by anyone. Conversely, they were rated worse by those who used the products less. However, the situation is different for the pragmatically focused products (PPT, PayPal). While PPT is very complex and extensive, PayPal does not fulfil all functions of the product category online banking, which means that for both products no significant influence by the usage frequency was found. The same applies to the external factor knowledge. For hedonically focused products (TikTok, partially BBB/Zoom), significant influences on the perceived UX could be detected. For pragmatically focused products (PayPal, PPT), on the other hand, no correlation was found.

The study can therefore be seen on the one hand as a recommendation to consider not only the purely product-specific UX factors but also the usage behaviour in the success concept of products. It should also be noted that the PQ should not be neglected for hedonically focused products and the HQ for pragmatically focused products.

The research has indicated that the choice of the measuring instrument is crucial in drawing accurate conclusions from the results. For instance, if a product's hedonic quality is a critical success factor, then it is imperative to use a dedicated scale to measure it. When a usability-focused method like SUS or UMUX-LITE is utilized, it may not be possible to identify variations in hedonic quality within the results.

Finally, an outlook on future work can be given. Firstly, the number of respondents available for our study was due to the division into the respective categories of the influencing factors relatively low. This is particularly problematic because the respondents were not evenly distributed across all categories of the influencing factors investigated. Therefore, some of the results are based on a small number of respondents, and they need to be corroborated with a more extensive range of products. Therefore, larger sample sizes will be relevant for future work.

Also, an extension of the range of products considered can shed even more light on the influence of external factors on the perceived UX of certain products and product categories as well as on the PQ and HQ in particular. Here, products of the same product categories (e.g., Amazon Prime as an alternative to Netflix, Instagram as an alternative to TikTok) would be conceivable.