Keywords

1 Introduction

What factors can be used to predict online trust and information behavior? Information shared via social media can influence decisions ranging from choosing which candidate to vote for, to which food truck serves the best breakfast tacos, to whether or not to vaccinate one’s children. This paper explores the potential impact of demographics and human values on online trust and information behavior. Specifically, we look at demographic categories such as gender, age, educational attainment, political leaning, and social media use. According to Friedman, Kahn, and Borning [1], “A value refers to what a person or group of people consider important in life” (p. 2). This paper operationalizes values at the middle level of Schwartz’s [2] four-level value inventory, including the following ten values: achievement, benevolence, conformity, hedonism, power, security, self-direction, stimulation, tradition, and universalism. This paper focuses on users’ clicking on hyperlinks as a specific type of information behavior.

This paper begins with a background focusing on prior research on the impact of demographics and human values on online trust and information behavior. The section after that describes how we collected and analyzed our data. The following section reports the results from the study. After that, we discuss the implications of our findings. Finally, we conclude by suggesting some future directions for research.

2 Background

The majority of Americans report getting at least some of their news on social media [3]. This is alarming due to the rise of ‘fake news’ stories being spread on social media platforms [4] by both social bots [5] and humans [6]. Coupled with the steady decline of trust in public trust towards mainstream media [7], understanding how people trust news sources online becomes critical. The difficulty of assessing online news sources is present across topics, including politics [4, 8], health [9], and science [10]. Since the general public and scientists’ view on science topics can have great variance [11] and science-related content on social media is highly popular [12], there is a need to focus on understanding social media users’ trust in science-related content online. Fleck [13] argued that scientific fact is invented and not discovered. This humanistic perspective on what we know to be scientifically true could also explain the persistence of fake news, particularly in science. However, Berghel [14] argues that the lack of emphasis on critical analysis in our educational system ill-equips the general population to make informed judgments about what to trust. This raises the question about how the public assesses trustworthiness of scientific news online, and whether demographic factors (e.g., education, age, gender) or values (e.g., what we hold to be important) influence how people trust science-related content on social media platforms.

The goal of this paper is to dig deeper into the role of demographic factors (e.g., gender, age, education attainment, political leaning, and frequency of social media use) and human values in influencing trust in different sources of online science news, including fake news, mainstream media, and scientific journals. We selected these factors as our focus due to their emergent importance in the research on trust in online news sources. This paper reports results from a larger study on factors that influence trust. A previous study on trust and social media and demographics found some significant relationships between age, gender, social media use and different concepts of trust, but they had a predominantly young participant sample and relied solely on a survey [15]. The findings of our study shed new light on these factors using an experimental design which allows us to measure trust more directly and specifically on real news items instead of concepts of trust.

There has been prior work that suggests that there is a relationship between gender and trust in online content. For example, Warner-Søderholm et al.’s [15] study on trust and social media found a significant relationship between gender and trust values: women had statistically higher scores on the trust values of “integrity” and “identification.” This could lend support to an earlier study that focused on how Chinese students assessed Wikipedia pages. That study found that women relied significantly more on validity when assessing trust whereas men focused more on accuracy and stability [16]. A study focused on how journalists source and trust information found that men had higher levels of trust in online news sources compared to women [17]. However, there is work that suggests there are no differences across gender [18]. Overall, findings regarding the effect of gender on trust in online news sources seem to be inconclusive, leaving room for further research.

The trope of the young outperforming the old in new technology is present in online news media research: age is a commonly discussed demographic factor in discussing trust in online news sources. While there is often higher use of online news sources among younger adults, there has been a substantial increase in older adults using the Internet and social media [3, 19, 20]. However, research seems to be mixed in regard to trust in online content. The 2017 Gallup/Knight Foundation Survey found that younger adults were more likely to have an unfavorable opinion of online news media compared to older adults [19]. This seems to suggest that young adults would be less trusting of online news media, however, several other studies seem to contradict this. In a study focusing on journalists’ trust of online content and in a study on privacy and trust on Facebook, young adults had higher trust levels in online content from social media [17, 18]. However, there are studies to suggest that trust could be low among both older and younger adults [21]. In general, findings regarding age and trust online also seem inconclusive, again demonstrating the need for further research.

Prior work on educational attainment also has had mixed results. Data from the Gallup/Knight Foundation survey supports the idea that those with higher education have higher trust in news media, other studies found that educational attainment did not impact interaction in online news [19, 22]. While these two studies are not identical to each other, there is a need to further explore the relationship between educational attainment and trust in online news sources.

Previous work indicated that political leaning certainly influences how people trust online news media. Liberals, overall, tend to have higher trust in the media, compared to conservatives [19]. Gallup’s data is supported by [23] study on news sources that found that liberals trusted a larger number of news sources compared to conservatives. Pew Research Center [24] also found that this trend continued to be true with liberals having higher trust in mainstream scientific outlets compared to conservatives. While literature does tend to provide a cohesive image, it is important to explore deeper into the role of political leaning.

Previous research also argues that the more people watch TV and read newspapers, the more likely they are to trust the media [25]. This trend seems to be consistent for social media outlets. With the majority of Americans getting some news from social media, frequency of social media use emerges as an important trust factor. An important distinction in measuring social media use is the difference between online skills (e.g., how familiar people are with using social media platforms) and the frequency of use of online news media (e.g., how often do they use it). There has been previous work that suggests that those who self-report a high skill level in using social media use social media outlets more frequently than those do have lower skills [26]. In this study, self-reported online skills were the main predictor of frequency of use of online news media, more so than gender, age, or amount of time the participant had been using online news media. This finding is supported by a study that found that those who had high knowledge of social media also had higher trust in it [17]. Interestingly, another study from Israel found that those who were less experienced online became more skeptical as they were exposed to more online news sources [27]. And in a recent cross-national study involving 44 countries, researchers found that people using the Internet to find political information had lower trust in the media over those who did not use the Internet to search for political information [28]. With the rise of social media being a dominant platform to get news, there is a need for further research into this factor.

With the rise in the polarization of politics and science, human values naturally enter the conversation. Martin [29] argues that opposing sides nearly always involve values as these topics often focus on topics of human life, economics, and other passionate topics. Values often play an important role in people’s behavior, attitudes, and sentiment [30] and can have a strong relationship with the development of opinions about polarizing topics [31, 32]. Price [33] argues, since it is impossible for us to be completely informed on a topic, we rely on our values, schemas, and group identifications to guide our attitudes and opinions on topics. Therefore, human values are arguably an important area to focus on to understand trust in online news media.

The goal of this study is to explore the impact of demographics and human values on online trust and information behavior. Specifically, this study sets out to answer the following three main research questions. First, do demographics and human values influence observed trust behavior? Second, do demographics and human values influence self-reported trust behavior? Third, do demographics and human values influence information behavior? We answer these questions using an experimental design that models how Twitter users review their timelines.

3 Methods

3.1 Study Design

We developed our scientific claims corpus by browsing two notorious fake news sites, InfoWars and Natural News [34]. We browsed articles in the “Science” section of Natural News and the “Science & Tech” section of InfoWars and hunted for links to open-access articles published in reputable scientific journals. For fake news articles that linked to an open-access scientific article, we then used Google News to search for open-access mainstream news articles that also discussed the same scientific claim and directly hyperlinked to the same scientific article. Our criteria for inclusion of a scientific claim were as follows: (1) the scientific article was publicly available; (2) a mainstream media news source directly linked to the scientific article; and (3) a fake media news source directly linked to the scientific article. To label the claim, the authors independently attempted to create a generic claim that corresponded with the titles and contents of all three articles, and then decided on the most neutral label. Thus, our corpus of scientific claims included a generic scientific claim, a “Scientific Journal” article from which the claim originated, a “Mainstream Media” article that covered the claim, and a “Fake News” article that provided a different perspective on the claim. It is important to note that this is an inherently biased sample of articles from InfoWars and Natural News, as many of the articles within the sites as a whole are not based on credible information. However, despite linking back to the open-access scientific article, the fake news articles tended toward a significant degree of sensationalization, with the mainstream media articles demonstrating arguably a somewhat lesser degree of sensationalization. Thus, for example, one of the scientific articles in the corpus, from Proceedings of the National Academy of Sciences (PNAS), was entitled, “Bottlenose dolphins can use learned vocal labels to address each other,” the corresponding mainstream media article from the Los Angeles Times was entitled, “Dolphins use signature whistles to call each other by ‘name,’” and the corresponding fake news article from Natural News was entitled, “Animal intelligence now irrefutable: Bottlenose dolphins call each other by name.” Here, the PNAS article uses the far more qualified phrase “learned vocal labels” while the article in the LA Times uses ‘name’ within scare quotes, and Natural News abandons the scare quotes and takes the finding to a somewhat more extreme level. Using this method, we were able to identify ten triads of mainstream media and fake news articles referencing the same scientific journal article.

Our instrument emulated Twitter’s timeline interface to approximate a modern online social media-based news feed. Each “post” in the mock timeline began with one of the ten scientific claims from our corpus. Two of these posts belonged to the “No Hyperlink” condition, and thus contained only the scientific claim with no hyperlink. The other eight posts included two with the corresponding “Fake News” hyperlink, two with the “Mainstream Media” hyperlink, two with the “Scientific Journal” hyperlink, and two with a “Hidden Hyperlink” (a shortened, obscured, version of the URL ultimately leading to the scientific journal article for the claim). Thus, there were a total of five conditions: “No Hyperlink (NH),” “Fake News (FN),” “Mainstream Media (MM),” “Scientific Journal (SJ),” and “Hidden Hyperlink (HH).” Thus, each participant was presented with a timeline featuring the same 10 claims but in a randomized sequence such that each of the five hyperlink types was represented twice in a given timeline.

We packaged the mock Twitter feed into Human Intelligence Tasks, or HITs, to be completed by “workers” on Amazon’s Mechanical Turk platform (“MTurk”). We chose MTurk based on a body of literature suggesting that MTurk offers higher accuracy of results, and greater demographic diversity in the participant pool than other commonly used samples such as college students or other Internet-based survey tools [35,36,37]. MTurk also afforded us with a pool of participants who presumably had at least basic computer literacy and were, therefore, likely to be familiar with the idea of obtaining news and other information from the Internet.

Each HIT consisted of the following: an informed consent screen; a gender-neutral version of the 21-item Portrait Values Questionnaire (PVQ) designed by Schwartz [38, 39] to measure the 10 human values as described in the Schwartz [2] Value Inventory (SVI); the mock Twitter timeline; a repeat of the same mock timeline but with a five-point Likert-type trust rating scale (ranging from “Don’t Trust At All” to “Trust Completely”) alongside each post; a 10-item questionnaire that presented 10 first-person assertions (eight unique assertions, one repeated assertion, and one to test workers’ attentiveness to the instrument) about workers’ perusal of information presented in the timeline; an open-ended question asking workers to describe how they arrived at their trust judgments; and a demographic questionnaire. In the 10-item questionnaire (see Table 1) we asked workers to indicate their level of agreement with each assertion on a five-point Likert-type scale (“Completely Disagree” to “Agree Completely”) to enable them to self-report their trust behavior. In the demographic questionnaire we asked workers to self-report their age, gender (“female”, “male”, and “other” with an option to specify their gender identity), political leaning & affiliation, number of years spent living in the US, and how frequently they used online social media along with names of the top three social networking platforms they used most frequently.

Table 1. Questionnaire items
Full size table

We collected data in two batches by requesting a batch of 100 HITs to be completed in April 2017, and a second batch of 150 HITs six weeks later in June 2017. HITs in the second batch contained two additional, open-ended questions at the end of the HIT asking workers for feedback and suggestions for improving any future versions of the study.

3.2 Data Analysis

We used nonparametric statistical methods; specifically, Mann-Whitney U to compare two independent samples [40] and Kruskal-Wallis H test to compare more than two independent samples [41]. Our choice for using these methods was driven by the fact that they minimize the number of assumptions one has to make about a given data distribution, while not increasing the likelihood of Type I errors or decreasing the power of the analysis [42]. To further decrease the likelihood of Type I errors based on computing multiple comparisons, we used the Holm-Bonferroni method to control the family-wise error rate [43]. To derive a combined trust rating for each post type per participant, we computed the mean of the two trust ratings they gave for the two posts they saw and rated per post type.

Across both phases of data collection, we eliminated 45 responses on account of non-completion of the HIT, incorrect answer to the attentiveness item in the questionnaire, and inconsistent responses (separated by a distance of more than one on the five-point scale) to the two repeated items in the questionnaire. This resulted in a final sample of 205 participants.

To simplify the analysis of demographic factors, we redefined the categories or levels within each factor to ensure all levels had relatively comparable numbers of individuals. For gender, the two most frequent responses were male (n = 116) and female (n = 88). To ensure the reliability of our analysis we dropped the data for the only participant who chose “other” as their gender identity. We categorized age data into three age categories: 18–29 years (n = 74), 30–39 years (n = 77), and ≧40 years (n = 54). We partitioned the educational attainment data into two categories: “Less than College” (those who chose either of “High School or Less”, or “Some College”; n = 106), and “College or Higher” (those who chose either of “College Graduate” or “Graduate Degree”; n = 99). For political leaning, participants reported their leaning on a five-point scale extending from “Conservative” to “Liberal”. We categorized those who chose the middle value on the scale as Moderates (n = 54), those who chose one of the two options on the conservative side of the scale as Conservatives (n = 51), and the rest as Liberals (n = 100). Finally, for social media use frequency, we created two categories: Light users (those who selected either of “Less Than Once a Week”, “Once a Week”, “Multiple Times a Week”, or “Once a Day”; n = 87), and Heavy users (those who selected “Multiple Times a Day”; n = 114).

4 Results

4.1 Observed Trust Behavior

Demographics.

This is the one comparison that has been previously reported [44]. Among the five demographic categories that we included in the study, only political leaning and social media use frequency revealed significant correlations with trust ratings. For political leaning, Kruskal-Wallis H tests showed significant differences in average trust rating for FN (p < 0.05), MM (p < 0.05), and SJ (p < 0.001) posts across conservatives, moderates, and liberals. Post-hoc pairwise Mann-Whitney U tests (with Holm-Bonferroni correction) revealed that, as compared to moderates, conservatives gave significantly higher trust ratings to FN posts (p < 0.05), and liberals gave significantly higher trust ratings to MM and SJ posts than moderates (p < 0.05). Liberals also gave significantly higher trust ratings to SJ posts when compared to conservatives (p < 0.01). Looking at social media use frequency, Mann-Whitney U tests showed that heavy social media users gave significantly lower trust ratings to FN posts than light users (p < 0.05).

Values.

Mann-Whitney U tests showed that people with high achievement displayed lower trust towards MM posts (p < 0.05) than people with low achievement. Compared to people with low universalism, those with high universalism showed higher trust in SJ posts (p < 0.01). Those with high security displayed higher trust for MM posts (p < 0.05).

4.2 Self-reported Trust Behavior

Demographics.

Among all demographic factors, only political leaning appeared to have any significant relationship with responses to the questionnaire on self-reported trust behavior. A Kruskal-Wallis H test revealed that liberals, moderates, and conservatives gave significantly different responses to item 8 on the questionnaire (“The content of the linked page influenced my trust in the post.”; p < 0.01). Post-hoc pairwise Mann-Whitney U tests (with Holm-Bonferroni correction) showed that liberals showed a significantly higher degree of agreement with the assertion in item 8 than moderates (p < 0.01).

Values.

We found that those who had high self-direction tended to agree with item 1 (“I noticed the presence or absence of links”) and item 4 (“The text portion of the post influenced my decision of whether or not to click the link”; both p < 0.05); and those who had high universalism tended to agree to item 1 (p < 0.01), item 5 (“The URL of the link influenced my trust in the post”; p < 0.05), and item 8 (“The content of the linked page influenced my trust in the post”; p < 0.05). Similarly, high security was also correlated with high agreement (p < 0.05) with item 1; high stimulation was correlated with high agreement to item 2 (“The text portion of the post influenced my trust in the post”; p < 0.01); and high benevolence was correlated with high agreement with item 7 (“I carefully reviewed the linked articles”; p < 0.01).

4.3 Information Behavior

Demographics.

Out of gender, age, educational attainment, political leaning, and social media use frequency, only age seems to have an impact on clicking behavior. A Kruskal-Wallis H test revealed a significant relationship between clicking behavior and age category (p < 0.01). Mann-Whitney U tests showed that people in the ≧40 year age category clicked on more hyperlinks (p < 0.01) than those in the 18–29 year category.

Looking at the number of hyperlinks clicked per post type, Kruskal-Wallis H tests revealed a significant relationship between click count and age category for FN (p < 0.01), MM (p < 0.05), and SJ (p < 0.01) posts. Mann-Whitney U tests (with Holm-Bonferroni correction) showed that for FN, MM, and SJ posts the ≧40 years age category displayed significantly higher (p < 0.05) number of clicks than the 18–29 years category. For SJ posts, however, the 30–39 year category also showed significantly higher number of clicks than the 18–29 year category (p < 0.05).

Values.

When we compared clicking behavior of the high and low value priority groups for each of the 10 values using Mann-Whitney U tests we found that only hedonism had a significant (p < 0.001) relationship with the total number of clicks: high hedonism was associated with a significantly lower total number of clicks than low hedonism.

Looking at the number of hyperlinks clicked per post type, we found that for HH posts high achievement is associated with a lower number of clicks (p < 0.05) than low achievement, and high self-direction is associated with a higher number of clicks (p < 0.05) than low self-direction. For FN posts, highly valuing security is associated with a higher number of clicks on FN hyperlinks (p < 0.05). High hedonism is associated with significantly lower number of clicks for each of FN (p < 0.01), MM (p < 0.001), and SJ (p < 0.01) post types.

5 Discussion

Our results suggest that the demographic categories of political leaning and social media use frequency bear potential to explain people’s trust behavior in relation to science news shared via social media. What is particularly noteworthy is that the influence of political leaning in shaping observed trust behavior is matched by people’s self-reported trust behavior. For instance, liberals not only gave higher trust ratings to mainstream media and scientific journal posts than moderates, but also reported a significantly higher degree of agreement with the assertion “The content of the linked page influenced my trust in the post” than moderates. The emergence of political leaning as a common thread in observed and self-reported trust behaviors is in line with the trends reported by the Gallup/Knight Foundation survey [19], and the Pew Research Center [24]. Our experiment therefore helps strengthen the case for further investigation into political leaning as a predictor or moderator of trust in online news about science.

We also observed that people’s trust behavior is influenced by their values: achievement and security seem to impact trust in mainstream media reporting of science news, whereas universalism seems to be directly related to trust in scientific journals as primary sources of science news. When it comes to users’ self-reporting of factors they believe influence their trust behavior, we found that people with high universalism were more likely to notice the presence of hyperlinks in social media posts, and their trust judgments were more likely to be influenced by the URL and content of the linked pages.

Our findings also shed light on the potential role values play in shaping people’s interaction with hyperlinks (and potentially other features of online information interfaces) on social media posts. The observed association between values such as hedonism, achievement, self-direction, and security with the number of times people clicked on hyperlinks helps add nuance to understanding the role of values in studying information behavior in general. It further underscores the importance of considering these core human characteristics both in the study and design of information systems.

The role played by values in turn may help shed light on the underpinnings of political leaning and its influence on trust behavior. Since values help shape people’s beliefs, attitudes, and sentiments [30], it is plausible that values also contribute in shaping people’s political leaning.

As with all studies, there are some limitations to the experimental design here. First, the data does not allow us to objectively separate the effects of the claim and the hyperlink on trust and clicking behavior. We do have self-report data along these lines, but asking people to assess the factors that influenced their decisions is much more subjective than observing would be. Second, we do not know if participants actually reviewed the contents of linked web pages after clicking. We know which users clicked on hyperlinks, but not whether or not they actually reviewed the linked site, and which aspects of the linked site they skimmed or read. Finally, based on this data it is challenging to determine if there are any cognitive ‘footprints’ that characterize trust formation and clicking behavior. Such analysis would best be performed in a usability laboratory.

6 Future Research Directions

As outlined by our findings above, we feel encouraged about the potential of human values, political leaning, age, and frequency of social media use to serve as reliable predictors of people’s information behavior online. The overarching question that will guide our future efforts is: how do people experience social media news feeds? Measuring their experience will enable us to understand how that experience shapes their trust judgments and information behavior vis-a-vis fake news.

One way to address the limitations described above would be to measure participants’ physiological responses to stimuli typically experienced when consuming information on social media. Employing eye tracking to collect data on participants’ visual fixations, saccades, and scanpaths can determine any correlations between their cognitive activity and trust judgment behavior. Fixation data would help to determine on which parts of the posts, or the timeline, participants were focusing. Saccades (the rapid movements between fixations) and scanpaths (a visual map of the fixations and saccades) allow for visualization of how participants brain scan the visual stimuli present in a (mock) social media feed as well as in actual websites. Eye trackers also record data on pupil size, which is a reasonably direct indicator of the cognitive effort exerted by the brain. In addition to eye tracking, facial recognition software can be used to record the movements of facial muscles to determine emotions displayed by participants as they interact with different stimuli.

Another refinement would be to use the Retrospective Verbal Protocol [45] to ask participants about the factors they believe influenced their trust judgments in a post-experience interview in which we will also ask them about factors that influence their trust in websites and social media posts in general.

Based on the questions raised by our current analyses, as well as the added capabilities provided by a usability lab environment, we propose the following preliminary hypotheses that could be tested in a usability lab context:

  1. 1.

    Long fixations on the hyperlink followed by a decision not to click on the hyperlink will be negatively associated with trust.

  2. 2.

    Participants who have negative facial expressions when fixating on a hyperlink, whether or not they click on the hyperlink, will have lower trust.

  3. 3.

    The explicit requirement to provide a trust rating will be associated with higher cognitive load than when just looking at the post.

  4. 4.

    Clicking on a hyperlink has a direct relationship with cognitive load (i.e., clicking is associated with high cognitive load, and not clicking is associated with low cognitive load).