1 Introduction

Instant messaging (IM) applications allow users to maintain interpersonal contacts, share opinions or exchange impressions. Through them, users can maintain relationships regardless of distance and reach people with whom a face-to-face meeting would be impossible. The number of IM applications available is constantly increasing. This phenomenon has been recognized by several researchers, and a lot of research is currently being done in this area [1, 2]. The number of entries resulting from a search for the term “instant messenger” on portals such as Google Scholar, ResearchGate.com or Scopus.com clearly shows this. Many questions arise, such as:

  • What methods should be used to assess the usability of instant messaging applications?

  • How to design, construct, and test a research tool?

  • Which variables to select?

These are only a few of the problems that a researcher who studies the phenomenon, which is the contemporary popularity of instant messaging, must find a solution to.

All instant messaging applications, regardless of the details of its functionality, are classified as IT software. The first known tool for testing the acceptance of an IT application was the Technology Acceptance Model (TAM). In the course of scientific development, many other methods emerged, such as the Unified Theory of Acceptance and Use of Technology (UTAT) and its variants UTAUT2 and UTAUT3 [3].

These models have already been used to measure technology acceptance of instant messengers. An example is research [4] using the theory of planned behavior (TPB), TAM, and the flow theory. Results demonstrate that users’ perceived usefulness and perceived enjoyment (TAM variables) positively influence their attitude towards IM applications, which in turn impacts their satisfaction. The results of [5] also support that perceived enjoyment and perceived usefulness are positively associated with user satisfaction. Issues such as the influence of technology acceptance model factors, social influence factors and demographic factors on instant messaging adoption in the workplace are presented in [6]. The researchers found social influence to be a more important factor in determining IM adoption than perceived usefulness and perceived ease of use. They also showed that gender and age did not impact the adoption of IMs. The acceptance of IMs in the workplace is studied by [7]. The author added an additional, new variable: “curiosity about other people” to the TAM model. The obtained results prove that perceived usefulness was not significantly important; however, perceived enjoyment, social norms, curiosity about other people, and perceived ease of use were all important prerequisites for IM usage outside the workplace.

2 Research Problem and Hypothesis

Technology acceptance models have been applied to many instant messaging applications, however, the variety of variables and constructs used by researchers is quite large (see next section for a comprehensive literature review). It is difficult to compare the results of studies from the same field when they use different models, variables, and measurement tools. We believe that technology acceptance research can benefit from identifying a common set of variables and operationalizing them in a unified questionnaire. Therefore, we present the first research problem:

  • Is it possible to identify the most robust technology acceptance variables and select questionnaire items that can be used consistently across different instant messaging applications?

The first research aim leads us to the second problem, which has never been fully addressed in the literature: technology acceptance models can be viewed as either fixed effect models or random effect models. The fixed effect view assumes that there is only one true technology acceptance and that, once identified, it can explain phenomena from a variety of domains (e.g., variable A explains behavioral intentions to use social media applications as well as blockchain adoption). This view is at the core of technology acceptance research and corresponds to the Davis F.D. conceptualization of TAM as a collection of variables that are general enough to explain the widest range of IT applications [8]. The same fixed-effect view is characteristic for the UTAUT model [3]. However, the recent meta-analysis results [9, 10] suggest that there is too much heterogeneity in the different technology acceptance models to grant the assumption of fixed effects. Variable A can explain well the intention to use social media, but not the intention to adopt blockchain. The need for random effects becomes clear. Random effects do not assume a constant effect that varies only slightly between different IT applications, but many effects. Random effects do not describe one technology acceptance that applies to all and explains both social media and blockchain, but one technology acceptance for a particular social media application and the second for a particular blockchain application (and they are certainly different). The existence of random effects is the reason (albeit often unconscious) researchers add new constructs for technology acceptance tailored to a specific IT application e.g., [11] added two new UTAUT variables to specifically explain technology acceptance of e-learning software).

However, the concept of fixed-effect technology acceptance may not be completely out of the air. For specific IT applications, such as the one we address in this paper, there is variability, but there may be hope for homogeneity. For example, WhatsApp might be perceived as a more useful messaging application (perceived usefulness) than Meta Messenger, but this variability might be generated by the underlying common effect for all messaging applications. Of course, this depends on the size of the variance and whether it is small enough to confirm the assumption of an underlying homogeneous effect.

We can formulate our second research problem as follows:

  • Can social media TAM variables be measured as fixed effects?

In other words, we want to select questionnaire items that are good indicators of TAM variables regardless of the type (brand) of social media messaging application. We can express this problem in a testable hypothesis:

  • H: Questionnaire factor loadings are reliably invariant across different social media applications.

In the first section, we present our approach to item selection and the conceptualization of new technology acceptance variables for social media research. In the second section, we test whether item invariance can be detected across different social media applications, confirming the fixed effect hypothesis at the variable level. Finally, in the appendix, we present a final set of robust items that can reliably assess technology acceptance of instant messaging applications under different circumstances.

3 Research Design and Methodology

In order to build the questionnaire for the pilot studies, six stages of design and analysis presented in Fig. 1 were identified.

Fig. 1.
figure 1

Research stages

Full size image

Work on examining the acceptance of technology in the case of messaging apps began with a literature review in this area (stage 1). The focus was on articles on technology acceptance research in social media, including messaging apps using the TAM or UTAUT models. A scoping literature search was conducted using the following electronic journal databases: Scopus, Science Direct, Emerald, IEEE, Springer, Taylor & Francis, Wiley, and Google Scholar. This scoping review, or scoping study, synthesized exploratory keywords aimed at mapping key concepts, types of evidence and gaps in research by systematically searching, selecting and synthesizing existing knowledge. We specifically searched for the following keywords and terms, including: “social media + TAM”, “social media+UTAUT model:, “Facebook+TAM model”, “Facebook+UTAUT model”, “Whatsapp+TAM model”, “Whatsapp+UTAUT model:, “Instagram +TAM model”, “Instagram+UTAUT model”, “Snapchat+TAM model”, “Snapchat+UTAUT model”, “Telegram+TAM model”, “Telegram+UTAUT model”, “Signal+TAM model”, “Signal+UTAUT model”, “TikTok+TAM model”, “TikTok+UTAUT model”, “Discord+TAM model”, “Discord+UTAUT model”, “Twitter+TAM model”, “Twitter+UTAT model”.

Table 1 presents a numerical list of the effects of the review of article databases in the studied area.

Table 1. Number of results according to combinative keywords search used for the methodology in electronic journal databases
Full size table

At length, we identified almost 28 7960 articles and conference papers in the first step of the search. To better focus our base, we filtered out articles published before 2010. From this original compilation of literature, we then identified articles that included survey-based research. After the initial analysis, around 600 articles were identified. We were interested in the variables used in the research and the questions that made up the questionnaires. In the database of the collected articles, particular attention was paid to review articles containing an overview of research into the application of technology acceptance models in the area of social media. The literature review was compiled, publications were systematically analyzed, using strategic and critical reading methods [12, 13]. We found the article by Al-Qaysi et al. [14], which served as a road map for the authors. This article is a review article identifying research based on the survey method in the field of social media research. It consists of the list about 60 articles, most of which used the survey method in the study. Some of these articles contained questions that authors asked to respondents. After a detailed analysis of the identified articles and with the use of the snowball method, we gained access to further studies using the questionnaire to measure the acceptance of technology in social media. On this basis, we created a database of 68 articles that we used to build the research tool. These articles were reviewed in detail and 21 of them were used to build the final tool.

Data gathering - questions

The systematic review mentioned in previous paragraph consists of 68 selected articles. To build the initial list of questions, we defined the following exclusion/inclusion criteria:

  • Is the article related to general available, popular social media application?

  • Does the article include the questionnaire with TAM/UTAUT related questions?

First step at the stage 3 of the pilot research design was to eliminate the articles without questions included in the paper and not related to popular social media generally available for users. As an example, articles related to network usage, web 2.0 and web-based communities were removed as there were not related to any software that can be utilized by the global society. The 21 articles related to social media tools were identified.

To build the final question database (stage 4) with TAM/UTAUT questions, all the 21 articles were reviewed. 825 questions related to 16 social media applications are identified. For each of selected items we captured details about the construct from UTAUT/TAM, social media name, article details and the question itself. All the captured details were grouped by the construct name – in total, we identified 87 UTAUT/TAM variable names (see Supplementary materials for a list of all variable names and items identified in the literature).

Created database was a subject of further analysis.

Building the final pilot questionnaire requires reduction number of questions and eliminate or merge the variables of UTAUT/TAM model, to keep the model sustainable and consistent. For this part of stage 4 of the design, the following exclusion/inclusion criteria were specified:

  • Is this question duplicated?

  • Does the defined software have messaging/communication functionality?

  • Is the question related to communication/messaging aspects between users (not marketing or announcements)?

  • Does the question fit into Likert scale?

The research is focused on communication aspect of social media, so the first step of selecting relevant questions was eliminating non-communication/messaging related tools. As an example, questions related to YouTube were eliminated, as text-based communication is not the main usage model of the application. Moreover, questions related to internal corporate blogs were removed because this kind of communicator is not widely available for general usage. On the limited number of questions (283), we looked for duplicated or similar questions and rephrase or remove them from the database.

Social media are powerful communication medium with high impact on social lives [15, 16] and it offers variety of use cases [22, 23], including e-commerce [19], marketing [2, 20], advertising [21], education and learning [22, 23], as well as exchanging messages and information between users [24]. To achieve the sustainable number of questions, we decided to select text-based communication between users as a focal point of the research. Questions related to Social Media Ads, online teaching, marketing, job offering were removed from the database. The last step of database clean-up was removing the questions which do not fit into the Likert Scale, for example, questions with Yes/No answers.

To create the final base for survey, the UTAUT/TAM constructs shall have the unified names [1]. Selected questions were grouped by the variables and variable names were reviewed. As a conclusion some of the variable names were unified, for example “Performance Efficiency” related questions were classified as “Perceived Usefulness”, “Satisfaction”, “Entertainment value”, “Perceived playfulness” was translated into “Perceived Enjoyment”. During the analysis, two additional constructs as model extension were identified and are used in the questionnaire - “Security & Privacy” and “Technology Attachment”. “Security & Privacy” questions were taken from other variables named “Trustworthiness”, “Online privacy” and “Intrusiveness tolerance”. “Technology attachment” is combination of “Affinity with computer”, “Technology-fit”, “Computer playfulness”.

Final list of the questions contains 86 items and contains 8 technology acceptance constructs:

  1. 1.

    Perceived Ease of Use

  2. 2.

    Perceived Enjoyment

  3. 3.

    Social Influence

  4. 4.

    Behavioral Intention

  5. 5.

    Attitude

  6. 6.

    Perceived Usefulness

  7. 7.

    Security & Privacy

  8. 8.

    Technology attachment

Building the questionnaire

Stage 4 of creating the questionnaire contained selection of the most used social media application, taking into consideration communication aspect of the media. Based on the information collected on the popularity of IM applications [25], we decided to qualify nine applications for the study. Selected applications which are includes in final version of questionnaire:

  1. 1.

    Meta Messenger

  2. 2.

    WhatsApp,

  3. 3.

    Instagram DM

  4. 4.

    Tik Tok

  5. 5.

    Twitter

  6. 6.

    Snapchat

  7. 7.

    Signal

  8. 8.

    Telegram

  9. 9.

    Discord

Research was designed to be taken in Poland, so all the questions were translated to Polish. The survey was configured and distributed using the Unipark online tool. Survey was available from 24.03.2022–24.04.2022.

4 Participants

97 respondents completed a designed questionnaire. The respondents were university students from northern Poland. Among the respondents, the majority are Meta Messenger users (92 out of 97). The second most popular messaging app is Instagram (74 of 97), followed by WhatsApp (56 of 97), SnapChat (36 of 97) and Discord (37 of 97). Other IM applications have user bases of less than 30 users, in our study.

5 Questionnaire Analysis

Data analysis scheme

To select items that are the best indicators for UTAUT variables, we created a data analysis scheme, which we divided into three stages: item analysis, exploratory factor analysis, and reliability analysis. Figure 2 presents the proposed scheme. We excluded messaging applications with a small user base (<30) due to the impracticality of conducting a factor analysis with samples smaller than 30 observations. The analysis includes only Meta Messenger, Instagram, WhatsApp, SnapChat and Discord. All results were calculated using laavan package in R programming language framework. Our data is available with supplementary materials.

Fig. 2.
figure 2

Data analysis flowchart

Full size image

Item analysis

In the first stage, for each of the 8 operationalized UTAUT variables and each of the 5 messaging applications, we conducted a classic item analysis. We selected three descriptive statistics that we considered key indicators of item goodness: standard deviation, range and kurtosis.

Standard deviation provides information about the variance in respondents’ answers. Items that have a small standard deviation do not discriminate between respondents, since most give very similar answers. Therefore, if an item’s standard deviation was lower than 1 (1 point on the Likert scale) or at least one messaging app, we removed this item from further analysis.

Most of the small standard deviations are for responses to items concerning Meta Messenger. We observed a very strong tendency to rate Meta Messenger positively as a messaging app. For example, item PU7, indicating a positive attitude toward the app, (Using the indicated communicator makes it easier for me to keep in touch with others) - has a standard deviation of 0.88 and a mean of 6.71 for Meta Messenger. It suggests a strong tendency of respondents to agree with this characteristic of this particular messaging app. For the other messaging applications, this tendency was weaker, and we considered it insignificant (e.g., item statistics for item PU7 and Discord are SD = 2.05 and M = 5.03). 13 items were removed from the analysis due to small standard deviation.

Range provides information about the spectrum of responses. Since we used a 7-degree Likert scale, we should expect a range equal to 6 for representative items. A range lower than 6 could indicate that an item does not capture the complete possible spectrum. For example, item SI7 (My friends also use the indicated messenger) - had a range equal to 3 for Messenger and Instagram, indicating that participants selected only responses between 5 and 7 (a strong tendency to agree). We found 9 items in our dataset that had a range below 6 and removed them from the analysis accordingly.

Kurtosis is a distribution measure. Distributions with large (positive) kurtosis are characterized by a concentration of responses around a single value. This is an undesirable characteristic for a well-discriminating item. For example, item SI3 (Most of my friends use the indicated communicator) - had a kurtosis of 30.57, suggesting an extreme concentration of responses around the mean. We removed 13 items with extreme kurtosis (above 10) from the analysis. Again, the majority of items with extreme kurtosis occurred for social influence variable and for Meta Messenger and Instagram messaging applications.

Exploratory factor analysis

In the second stage, we conducted an exploratory factor analysis for each of the 8 variables and each of the 5 messaging applications. Exploratory factor analysis measures which items are the best indicators for the UTUAT variables. The basic statistics of factor analysis are factor loadings. We can interpret a factor loading as a correlation between an item and a latent variable (UTAUT variable). The larger the factor loading, the better the item functions as a statistical indicator. Due to the small sample size for some messaging applications (SnapChat and Discord) and the ordinal nature of our variables (Likert scale), we decided to use a non-parametric variant of exploratory factor analysis with a polychromatic correlation matrix and WLS as an estimator.

For each of 5 messaging applications, we conducted 9 exploratory factor analyses (one for each variable), giving us 45 factor analysis results. In order to quantify the association strength between an item and the UTAUT variable, we decided to aggregate factor loadings for 5 messaging applications using a factor loading weighted average. We used the proportion of users in the sample as weights (e.g.: Messenger has a weight of 92/97–0.948, WhatsApp 56/97–0.577). For each variable, we selected 4 items with the highest factor loading and lowest weighted factor variance (homogeneity).

Reliability analysis

In the third stage, we measured reliability of selected items. For each 8 variables, we calculated a reliability coefficient [26] using weighted factor lodgings from the previous stage. The reliability coefficient is formulated as:

$$\omega = \frac{{\left( {\sum {\lambda_{i} } } \right)^{2} }}{{\left( {\sum {\lambda_{i} } } \right)^{2} + \sum {(1 - \lambda_{i}^{2} )} }}$$

where is an average-weighted loading.

Table 2 presents the reliability and descriptive statistics for the final selection of items.

Invariance results

Table 2 shows that the variances of the weighted factor loadings are almost nonexistent. This is true only for the final selection of items. Other combinations of items, rejected by our analysis, would have resulted in significantly larger variance. This proves that invariance can be achieved at the item level but does not mean that it is not possible to create questionnaires that do not measure technology acceptance homogeneously across different IT applications. For this reason, we encourage future researchers to use our questionnaire, as it is consistent with the fixed effect hypothesis.

Table 2. Factor and reliability analysis results
Full size table

6 Conclusions and Further Research

Applying an extensive literature review, we were able to select items that have reliably measured technology acceptance variables in previous works on social media messaging applications [1, 27, 28]. In addition, we combined these items into 7 variables that best reflect technology acceptance of social media [29]. We further added the eighth variable, reflecting the user’s psychological dependence on a messaging application. These items were analyzed extensively to:

  • select the best items that can be used by future researchers of social media technology acceptance,

  • prove that technology acceptance is invariant at the construct level and within a specific IT application (social media messaging).

The final product of our research is a questionnaire that can be used by other researchers to investigate technology acceptance of social media applications. Our questionnaire unifies the common constructs of technology acceptance in a comprehensive and reliable framework. It guarantees that the measured variables are invariant and satisfy the fixed effect hypothesis.

Future researchers can use our questionnaire to deepen the understanding of social media usage and to test the structural relationships between the proposed variables. The problem of fixed/random effects in technology acceptance models is far from solved. We have demonstrated that the construction of an invariant questionnaire is possible. Further research should investigate whether the TAM variables are invariant at the structural equation (SEM) level. To prove the invariance, a technology acceptance model (TAM) of major social media applications must be created (Facebook-Messenger, Instagram, WhatsApp, Twitter, etc.). Besides model invariance, researchers can also use technology acceptance for comparative SWOT analysis and product design analysis of individual IM applications. Such analyses can be useful in determining flaws and desirable functionalities of those applications. They may also provide insight into which TAM variables contribute the most to the market position of a certain social media.