Keywords

1 Introduction

In times of digitization, the Internet plays a dominant role in people’s lives. In addition to the use for communication and entertainment reasons, the Internet is a medium for information search. The development of digital information is increasing. Day by day, the available information increases in quantity. Information about e.g., places, people, hours, or news are just a few search topics. One of the most sought areas is health issues [1]. Above all, due to a new awareness of health and lifestyle (i.e., quantified self) and the development of informed patients, information is more relevant than ever. In addition to being informed only about health issues, people also take information as the basis for decisions about treatment or the intake of medicines [2]. Digital health information offers many benefits. Health information is always and available everywhere. It gives many people access to medical information [3]. People can actively participate in health issues and even collaborate with other people who are working on the same topic [4]. However, disadvantages of so much information circulating on the Internet are also present. Much of the information found online is non-serious or outright false and not recognized as being false or outdated [3]. Therefore, there may be psychological or physical consequences that may be incurred by relying on e.g., medical advice or false intake of medicine [5]. Therefore, it is a major challenge for health information researchers to evaluate the quality and credibility of websites [6]. People focus on different criteria for assessing information as correct [7]. There is a growing need to understand how this information is accessed and how it is used. What criteria are important for people’s decision to trust the information? On the other hand, how diverse are the users? Which user prefers what kind of presentation? In this study we examine this kind of question. The aim of the study is to find out and to understand to what extent the media representation of health websites and the degree of the described disease play a role in their assessment. The study also aims to understand what kind of user characteristics affect the site’s rating. This article—which extends upon earlier work [8]—is structured as follows: After this introduction, the current state of health information on the Internet, quality of health-related websites, credibility factors of health-related information, e-health literacy, and privacy concerns considering general search behavior are presented. Section 3 describes the research questions that guide the study as well as the research methodology. Section 4 presents the focus group approach regarding the generation of credibility factors, whereas Sect. 5 describes the questionnaire approach with respect to the research questions. Section 6 discusses the results and guidelines for website developers. Section 6.1 concludes this article with a brief discussion of the limitations of this work and an outlook on other research questions.

2 Related Work

In order to answer the questions that we address, we must first understand the state of the art in digital health information (see Sect. 2.1). Further, it is necessary to understand how quality of health related websites can be described (see Sect. 2.2). And since users perceive these quality aspects differently and turn them into credibility factors for health-related information (see Sect. 2.3) understanding what drives these differences is crucial. One core skill that determines the quality of this process is the concept of e-health literacy (see Sect. 2.4). Additionally, users have different privacy concerns and needs for privacy, which also impact search behavior (see Sect. 2.5). The following sections aim to provide an overview of how these topics interrelate to each other to motivate our research questions.

2.1 Digital Health Information

Information about health is often searched for online. More than 70% of people search the Internet for such information [1]. Most people conduct searches about their own health problems. Key topics include disease symptoms, prognoses, and treatment options [9]. In a study by Kienhus et al. [10] 61% of users who examined the impact of online patient search on patient-physiological interaction reported that the information they sought affected their own health. This finding underscores the need to understand people’s reasons for evaluating information as trustworthy whenever it affects their health.

2.2 Quality of Health-Related Websites

Health information is often accessed on the Internet. Although many tools and policies already exist to obtain high-quality information [11], health-related information and its quality still fluctuate widely across the Internet [12]. Aspects of lack of information quality reveal a wide range of information that is not serious, out-of-date, or contain incorrect information. In addition, websites often serve as a platform for advertising rather than a platform for evidence-based sources. One of the biggest challenges for information seekers is therefore to evaluate the present information. But not only the content, but also the presentation of information such as layout, structure, images, etc. are aspects that influence the evaluation by the user.

2.3 Credibility Factors of Health-Related Information

The overdose of information that appears online when looking for information about health issues on the Internet is overwhelming. However, people have developed their own search behavior and when it comes to, e.g., health information sites, people focus on specific aspects. Which so-called credibility factors make digital information useful and trustworthy? Many studies have been conducted regarding this phenomenon. Eysenbach and Köhler [13] report that, for example, references, information on the latest update, as well as information on authors and images become important credibility factors of credible websites. In addition, information on alternative treatment options and side effects on health-related websites are additional credibility factors [14]. Benigeri and Pluye [15] have developed an approach to describe support criteria for the quality assessment of health-related digital information. Although there are already numerous catalogs of credibility factors, the quality rating still varies. It seems that user diversity strongly determines the importance of various aspects. Barnes et al. [16] found that, e.g., the extent of personal involvement has an impact on the evaluation of information. Less involved people seem to focus more on layout than content and timeliness, as is the case with more health-related stakeholders. In addition, younger people focus more on website layout [17] than older people who are more interested in references [18]. Our study aims to review these results and to identify more specific aspects taking into account user diversity.

2.4 eHealth Literacy

In addition to objective criteria such as the above-mentioned credibility factors, subjective ability factors play an important role in the evaluation of digital health information. Literacy is a very important aspect. People who are able to read and write have higher education, integrate and participate more easily in social life, and are able to understand and execute a greater degree of control over everyday events [19]. The term “eHealth literacy” describes the ability to search, find, read, understand, and evaluate health information from electronic sources. It means that people have the ability to apply the knowledge they have acquired and to address or solve a health problem [20]. Greater health literacy correlates with better health outcomes. Health literacy influences the use of health care, patient-physician relationship, and self-care [21]. It is obvious that the evaluation of digital health information differs strongly due to the individual differences in competence. To find out about the phenomenon of health literacy, it is integrated into our study.

2.5 Online Privacy

Hand in hand with the search for health-related online information goes the growing concern of the users’ online information privacy. Whereas the online search for health-related topics allows user to ask sensitive or detailed question without the risk of facing judgment or stigma [22], in the same time people start worrying about their personal information which they leave behind when looking for some specific health issues. As personal information becomes more accessible, users worry that institutions and especially insurance companies misuse the information that is collected while users are online [23]. However, the level of awareness varies just as much as different users are. To find out to what extent users looking for online information also give additional thought to their privacy, we considered to include the phenomena of privacy concern and need for privacy in our study.

3 Question Addressed and Logic of Empirical Procedure

This paper raises the question of how digital health information is judged by its recipient and how it varies with the severity of the disease. In particular, the focus is on the perceived credibility of various media representations of websites, which are also influenced by two different degrees of severity of the disease. The purpose of the paper is to identify credibility factors that affect the recipient’s attitude of trusting a health information website. Guidelines for site developers are presented based on the results. In order to identify, evaluate and quantify these factors, a two-way multi-method approach was chosen. In a first step, the data was collected qualitatively according to focus groups. Three focus groups were run with three different age groups. Since the methodical approach of the focus groups intended to collect different opinions of people’s point of view, very general questions were guiding the group discussion:

  1. 1.

    Where do you look for health-related information?

  2. 2.

    Which factors make information appear credible?

Based on the results, a questionnaire was developed and the data was collected quantitatively. Questions guiding the questionnaire approach were:

  1. 1.

    What are the key trust elements of a website that presents health information in general? Do age and gender influence the evaluation of credibility factors? Are the credibility factors different in relation to the severity of the disease?

  2. 2.

    To what extent does the media representation of a health website and the degree of the disease play a role in the opinion of the user?

  3. 3.

    To what extent do user factors such as age and gender influence the assessment of health information of varying severity?

Fig. 1.
figure 1

Overview of research process showing both qualitative and quantitative measures to address our researach questions.

Full size image

In Fig. 1, an overview of our research process is depicted. This method section is structured according to our study process. First, the results of the focus group studies are presented. Second, the development of the questionnaire will be introduced. Then the selected statistical methods and the collected samples are described.

4 Generation of Credibility Factors - The Focus Group Approach

The focus group approach aimed at gaining deeper insights into people’s search behavior when looking for health related information. Moreover, the purpose was to identify and discuss people’s ideas of factors which make health-related information on websites appear credible.

4.1 Method

First, participants were made familiar with the topic and the idea to gain knowledge about the individual favored kind of representation of health related information on websites. A general question (Where do you look for health-related information?) was raised in the beginning. Participants were encouraged to brainstorm about sources they use when informing themselves about health related information. As an “icebreaker” or rather stimulus for the discussion, the moderator shared a personal and comprehensible experience she had made recently. She described the situation of a “nervously tickling sensation” in her eye and how she started to search the Internet for reasons and for ways how to get rid of it. Thereby, she found plenty of different information partly with different opinions. Which one was she supposed to trust? In a free discussion participants started to share similar experiences. The different source types mentioned were written on paper and collected on a pin board. In a next step participants were introduced to a more serious health topic in form of a persona. Participants’ were shown a picture of a mid-40-year old woman and asked to put them self in the position of the described person who just recently received a diagnosis of breast cancer. The method of the persona was intended to help the participants to identify themselves with her and start thinking from her point of view [24]. Participants were encouraged to consider where they would look for information in place of the persona. Again, all newly mentioned mediums and sources were added to the pin board. In a further step, participants were asked (Which factors make information appear credible?) to gather factors which make health related information on websites seem trustworthy. The mentioned factors were collected and rated due to their importance. In the end a short questionnaire was applied. Items were taken and used from related work [20] and had to be answered on a 6-point Likert scales. The duration of the focus group was about 90 min.

4.2 Sample Description

The focus group was conducted with 17 participants in total but split into three sessions. The sample was composed of 13 female and 4 male participants with an age range from 14 to 69 years (\(M=44.8\), \(SD=0.44\)). One group with 5 participants was between 14 and 19 years old. The second one (\(N=6\)) encompassed 30 to 54 year old participants. The third group (\(N=6\)) contained 55 to 69 year old participants. The professional activities of the participants were very broadly diversified from students to employed to retired people. In general, participants have assessed themselves as very healthy with \(M=4.47\) on a six-point scale (\(SD=0.94\)). The evaluation with regard to their individual health literacy turned out to be average (\(M=3.82\), \(SD=1.08\)).

4.3 Results

Data of the qualitative focus group were analyzed descriptively using qualitative data analysis by Mayring [25]. Further, we used the tidytext, wordcloud, and tm package [26,27,28] to quantitatively analyze the transcripts from the focus groups in R and RStudio. Textual data was gathered from manual transcripts in pdf-files. Individual utterances were separated to allow individual-based analysis of data. Stopwords were removed using the stopwords package. We further removed numbers, punctuation signs, and an additional list of non-standard stopwords manually added after going through the full tokenized list. Since all transcripts of the focus groups were in German, word data was translated automatically using the Yandex-API after tokenization. All visualized data is “sanity-checked” after translation to fix possible errors of ambiguous translation. For sentiment analysis we used the SentiWS Dataset [29]. We use a bag-of-word model for sentiment analysis and manually correct for negations. We relied on this simple model of text-analysis as many of the utterances in live discussions have incoherent grammar and interjections. More advanced techniques of analysis fail to identify structure in such utterances. Therefore, we invested more effort in manual correction.

Fig. 2.
figure 2

Top 200 terms mentioned across all focus groups (N = 17).

Full size image

Do the Different Age Groups Differ? When we look at what the different participants mentioned during the focus groups, we see that the topics of highest interest include things like the Internet, whether one could ask questions, was there support from doctors, but also properties like easy usability, relatable examples, and images were important (see Fig. 2).

When we look at the differences between the different age groups and the topics that were relevant to the individual groups, we also see an interesting picture (see Fig. 3). While younger participants focused on ease of use and meta-data (e.g., was the date on the website shown), medium aged participants were discussing about individuality of cases, the verification of topics by doctors, inviting imagery, as well as official seals of approval. Interestingly, older participants focused their discussion around family topics, self-help groups and were mostly concerned about the imprint of a website, when it came to judging credibility.

Fig. 3.
figure 3

Top 100 terms that were different between focus groups.

Full size image

Which Factors Make Information Appear Credible? Using correlation analysis in the term-document matrix, where documents are individual utterances, we can identify, even in a bag-of-word model what words are mentioned in conjunction frequently. Using this technique we can see, for example, what other words are mention in conjunction with the word “credible” to identify what factors influence credibility of websites. By going back to the original utterances. In Fig. 4 we see that factors such as expert opinions, certainty, experience as well as “luminaries”Footnote 1 play a role in credibility judgments.

Fig. 4.
figure 4

Top 15 terms that were associated with the term “credible” (N = 17).

Full size image

Since the use of source was stressed by the participants in many of the discussions, we wanted to see what other topics were mentioned along the term “sources” (see Fig. 5). Participants stress, when talking about sources, that comprehensibility and sources in their mother tongue (i.e., German) are helpful. Sources are used to explain and evidence and participants prefer sources where the authors are reputable scientists and doctors. Some even mentioned that video-based explanations show the quality of a website and were interested in video-conferences with doctors.

Doctors were mentioned quite a few times during the discussion. Participants were unsure whether digital information alone could ever be a solution to medical questions and wondered, whether in serious matters questions could be addressed to real doctors by email (see Fig. 6). The strategy was mostly to combine the benefits of online information with the personal relationship with a doctor. Participants are aware that going to the doctor takes time, but creates highly individualized information, but Internet search provides immediate and in-depth feedback.

Fig. 5.
figure 5

Top 15 terms that were associated with the term “source” (N = 17).

Full size image
Fig. 6.
figure 6

Top 15 terms that were associated with the term “doctor”. The term day occurs twice in the data-set due to translation (N = 17).

Full size image
Table 1. Three utterances with lowest and highest summed sentiment.
Full size table

Summary of Qualitative Findings. Overall, we found that scanning transcripts of focus groups using natural language processing utilities is helpful in looking for topics and relationships among topics. Still, the informal nature of such experiments creates language unfit for fully automatic evaluation and requires both manual and automatic effort. The sentiment analysis (see Table 1) requires special attention, since here a simple “not” or irony can invert the intended sentiment. But using strong sentiments as an indicator for further investigation proved helpful in finding statements, where participants made judgments about the topic at hand, which in turn were useful in factor generation. From the data we found that aspects of privacy and health literacy, but also website design and comprehensibility were important to users, focusing the research effort in the second phase of the study.

5 Validation of Credibility Factors and Media Representation - The Questionnaire Approach

To quantify the results of the focus group discussions with a larger sample, a questionnaire was developed. Results of the focus group and other criteria that can influence the credibility of health websites, such as age, gender and online privacy aspects, were taken into account and integrated into the questionnaire development. The survey consisted of three parts. Starting with demographic factors in part one (age, gender, education level, health status), variables regarding the person were assessed in a second part. These included the users’ privacy attitudes such as privacy concerns and need for privacy. E-health literacy was also surveyed in part two. Part three examined the users’ attitude concerning the different website scenarios with respect to the severity of disease. Credibility factors as generated results from the previous focus groups were also considered.

5.1 Part 1: Demographic Data and Further User Factors

The first part of the questionnaire assessed age, gender, sex, highest education level, current activity and health status. Moreover, general familiarity with the Internet usage (For how long have you been familiar with the usage of Internet? had to be answered on a 6-point Likert scale. The individual points were (1) not at all, (2) less than one year, (3) 1–2 years, (4) 2–5 years or (5) 5 years and more) as well as general Internet activities were collected (“How much time do you weekly use for following web activities in average?”; answering scale: no time, 0–30 min, 30–60 min, 1–2 h, 2–4 h, 4–8 h, 8 h and more). At last, usage frequency of information sources (“How often do you use the following information sources when informing yourself about health topics?”; answering scale: daily, 2–3 times a week, 1 time a week, monthly, 2–3 times per year, less) regarding health topics, usability of online sources (Searching for health information, how helpful do you rate the following sources? search engines, platforms, forums, chats, websites. Each source had to be assessed on a 6-point Likert scale from 1 = not helpful at all to 6 = very helpful) as well as interest regarding health topics were assessed (“For me, information about a healthy lifestyle/reason for cold/reason for sickness/diagnosis of cold/diagnosis of sickness/therapy of cold/therapy of sickness/medical treatment and physicians...are interesting.”.

5.2 Part 2: Need for Privacy, Privacy Concern and eHealth Literacy

Six items assessing need for privacy were taken from literature [30, 31] and had to be answered on a 6-point Likert scale from 1 = I do not agree at all to 6 = I totally agree. The items are listed in Table 2.

Table 2. Items for privacy concerns, need for privacy and ehealth literacy.
Full size table

An overall score was calculated out of the respective items after having recoded the negative items and after having checked the scale reliability (Cronbach’s \(\alpha = .803\)). Items collecting information about privacy concern were taken or adapted from different authors [30, 32,33,34,35] and had to be answered on the same 6-point Likert agreement scale as above (see Table 2).

Only two items (1 & 3) reached a sufficient Alpha value with (\(\alpha = .721\)) and were summed up to an overall score. The eHealth Literacy Scale (eHeals) by Norman and Skinner [20] was taken as an instrument that measures computer skills with health literacy skills. In Table 2 the eight items are listed. They had to be assessed on a 6-point Likert agreement scale.

5.3 Part 3: Website Scenarios

To find out in how far information about health topics are perceived and assessed on websites, three existing electronic health websites were chosen by authors and were arranged to fictive collages without naming the website brand. These developed websites could be distinguished according to its complexity of content and preparation of site. Regarding complexity, a website was stated to be complex when the information was more detailed and the layout contained more sub-units. At last, one website with low content and low representation was build (LowRep), one with a middle degree of complexity (MidRep) and a third one with a very high complexity (HighRep). Furthermore, two diseases with different degrees of severity were chosen. For a marginal but still serious disease hay fever was described. Breast cancer was taken as an example for a very severe and life striking disease. Participants were asked to look at the website leisurely and report their impression. Therefore, they were asked to rate 5 items afterwards on a 6-point Likert scale (1 = not at all to 6 = yes, in any case), shown in Table 3.

Table 3. List of website assessment items.
Full size table

The articles were analyzed with regard to the different website types using a factor analysis. Two factors have been identified. The first contained item one, two and three, which asked about the benefits, comfort of information and credibility (\(\alpha = .840\)). Resulting, a scale was build and called assessment scale. The second factor consisted of the items four and five asking about if more information is desired or a consultation with a physician is still wanted. Cronbach’s Alpha did not allow to calculate a second scale (\(\alpha = .571\)). Overall, 6 websites were represented; three for each disease (3 websites \(\times \) 2 diseases). To avoid fatigue effects, the order of websites was randomized.

Credibility Factors. After a “disease set”, namely three website versions, the participants were asked with which criteria a website can be classified as trustworthy. Therefore the participants had to personally name five out of 18 criteria as being the most important ones. The choice of criteria resulted from the focus group discussion. All mentioned criteria were used here in the questionnaire for validation (e.g., publication date, source, quality seal, etc.). The whole study was run within the framework of a bachelor thesis in summer 2017 in Germany. Data was collected through the personal and professional surrounding from the candidate of the final paper as well as from the authors online and in a paper-pencil form, enabling older people to participate as well. Participation was voluntary and was not gratified. Completing the questionnaire took about 20 min.

5.4 Sample Description

Demographic Data and Further User Factors. The survey was completed by \(N=184\) participants. 40% male and 60% female participants took part in the study. The sample reached an age range between 17 and 79 years (\(M=43.5\) years, \(SD=15.7\)). For an age comparison regarding different items, the sample was split by median into three age groups: 61 participants fell into the so called “digital natives” group (<29 years, 40 women and 21 men), 62 participants were assigned into the “digital immigrants” group (between 30–54 years, 37 women and 25 men) and 61 fell into the “silver surfers” group (>55 years, 33 women and 28 men). 37% of the participants hold a university degree and 29.9% a university entrance diploma. Moreover, 17.8% completed a higher education and 10.3% hold a secondary school certificate, indicating the heterogeneity of the sample’s educational level. Most of the participants (36.4%) allocated their current activity in the commercial area, 19% in a technical area, 16.3% allocated it to the social field, 7.1% to a medical field, 3.3% to a artistically field and 17.9% allocated their current activity to other areas. In general, the sample constituted a rather healthy group with \(M=4.2\) (\(SD=0.85\); 6 points max.). Table 4 portrays the demographic characteristics.

Table 4. Demographic characteristics of aggregated sample (N = 184) [8].
Full size table

When asked about familiarity with Internet use, the sample reported that it was quite familiar (\(M=4.93\), \(SD=0.4\)). The maximum duration of Internet activities such as reading newspapers (\(M=2.53\), \(SD=1.59\)), posting in newsgroups (\(M=2.12\), \(SD=1.28\)), receiving information about products (M = 2.83, SD = 1.17) or buying products (\(M=2.25\), \(SD=0.97\)) was limited to an average duration of 0–60 min per week. When asked how often participants use certain sources for information on health topics, the Internet was most frequently use with a monthly use (\(M=4.44\), \(SD=1.21\)), followed by relatives (\(M=4.44\), \(SD=1.15\)). Doctors (\(M=5.06\), \(SD=0.66\)), medical journals (\(M=5.49\), \(SD=0.9\)) or self-help books (\(M=5.51\), \(SD=0.89\)) were mentioned, which are seen or used 2–3 times a year. In addition, search engines (\(M=5.52\), \(SD=1.17\)) were described as the most helpful, followed by websites (\(M=4.22\), \(SD=0.98\)), platforms (\(M=3.41\), \(SD=1.24\)), forums (\(M=3.12\), \(SD=1.21\)) and recent chats (\(M=2.44\), \(SD=1.09\)). It was also of interest what kind of information the participants were looking for on the Internet. Information on a healthy lifestyle (\(M=4.13\), \(SD=1.22\)) followed by information on the treatment of serious diseases (\(M=3.85\), \(SD=1.17\)). Information on medical treatments (\(M=3.85\), \(SD=1.26\)) and doctors (\(M=3.75\), \(SD=1.4\)) were reported before causes (\(M=3.79\), \(SD=1.22\)) or diagnosis of diseases (\(M=3.61\), \(SD=1.26\)). The least important search terms among the health information were given due to a cold (\(M=2.6\), \(SD=1.27\)) and a cold diagnosis (\(M=2.49\), \(SD=1.14\)).

Need for Privacy and Privacy Concerns. The participants reported that they were a little concerned about their privacy (\(M=3.93\), \(SD=0.76\)). The results on the need for privacy scale reflect similar results with \(M=2.82\) (\(SD=0.61\)). A significant age difference was observed in the privacy concern attitude (\(F(2,170)=6.19; p=0.003\)). Here, older participants showed the highest concern regarding their online privacy and disclosure of health data with a mean of M = 4.10/6 points max. (\(SD=0.83\)) in contrast to the youngest group with a mean of M = 3.66/6 points max. (\(SD=0.73\)).

e-Health Literacy. The health literacy level was averagely high with \(M=3.84\) (\(SD=0.79\)). In this context, a significant age (\(F(2,\,152)=4.01;\;p=0.020\)) and gender (\(F(1,\,152)=5.05;\;p=0.026\)) effect could be detected. The ability to search, find, read, understand and evaluate health information from electronic sources was rated significantly higher by the youngest group (\(M=4.06,\;SD=0.7\)) than by the oldest group (\(M=3.59\), \(SD=0.85\)). Female participants (\(M=3.96\), \(SD=0.87\)) rated their competence to evaluate digital health information higher than male (\(M=3.67\), \(SD=0.63\)) participants.

5.5 Results

All subjective measures were evaluated on six-point Likert scales. The data were analyzed quantitatively using Pearson correlations, ANOVA with repeated measurement and MANOVA. The level of significance was set to \(\alpha = .05\). This means that significant findings can occur in 1 out of 20 such studies, even if the effect is not present.

5.6 Validation of Credibility Factors

The presentation of the results is guided by the research questions and is structured as follows: First, findings concerning the credibility factors of a health information website are presented. Secondly, the assessment of websites types (low, middle, high) regarding its complexity and its content (light disease vs. serious disease) will be outlined. The section closes with findings about the impact of user diversity regarding the assessment of websites.

Factors Influencing Credibility.

In order to identify credibility factors that affect recipients’ attitudes to rating a website with health information as trustworthy, participants had to identify five out of 18 criteria that were rated as personally most important. As most important comprehensibility was mentioned, followed by objectivity of the information, clear structure of the website reference and the indication of negative side effects or risks. Factors with low credibility were links to other websites, access to forums or chat rooms or images of authors. Considering the severity of diseases, the picture is different. Table 5 shows the results. If one compares the five most frequently mentioned credibility factors of a mild and a severe disease, it turns out that four aspects are the same only with a different significance. For example, comprehensibility is assessed as the most important aspect of health-related information for both sides. Other factors mentioned differ due to the severity of the disease. While the clarity of a website plays the second most important role for the health-related information of a mild disease, objectivity is mentioned in the second step. Level five contains information on authors as a credibility factor for a mild disease. In contrast, the date of publication is assessed as another important credibility factor for the digital health information of a serious disease. Taking into account the three different age groups, the factors mentioned remain the same, only the order of the factors mentioned differs slightly.

Table 5. Five most important assessed credibility factors of health-related websites with different disease contexts in % (N = 184) [8].
Full size table

Assessment of Different Websites. In order to find out to what extent the media presentation of a health website and the severity of the disease play a role in the user’s assessment, a repeated measure was calculated. Looking only at the media presentation of all three websites without the content of diseases, no significant difference could be found (\(F(1.82,\,300.57)=2.57,\;p=.084\)). Nevertheless, the most complex version was rated best (\(M=3.49,\;SD=0.77\)), followed by the second complex version (\(M=3.44,\;SD=0.78\)) and finally the version with the least complexity (\(M=3.34,\;SD=0.88\)). Interestingly, significant differences were found including the different severity of the diseases (\(F(3.7,\,581.5)=5.75,\;p<.01\)). Since Mauchly’s test showed that the assumption of sphericity was violated (\(\chi ^2(14)=113.48,\;p<.01\)), the Greenhouse Geisser corrected tests are reported (=.74). For the less severe disease version two achieved the best rating (\(M=3.64,\;SD=0.87\)). In contrast, of the three different website versions, version three was rated best for the more life-threatening disease (\(M=3.44,\;SD=0.9\)).

Fig. 7.
figure 7

Assessment of website regarding if more information is wished for and if a physician wants to be seen after having seen website. Error-bars denote standard error [8].

Full size image

The results suggest that evaluating the complexity of websites always plays an important role when describing a particular disease. More serious diseases are preferably read on the most complex website, as opposed to a mild disease. In our case, the participants liked to read information about a mild illness in a more unusual way. Significant results were also found (\(F(4.42,\;680.15)=25.72,\;p<.01\)) when asked whether, after seeing the website together with the nature of the disease, participants would like to seek further information or see a doctor. Figure 7 shows the different characteristics. The desire to get more information and to see a doctor increases with the severity of the disease. The evaluation of website versions shows a different picture. The most complex website seems to provide better information in the case of a mild illness than in the case of a serious illness in contrast to the other versions. However, the less complex website versions one and two seem to provide satisfactory information about the more serious disease than about a mild one. The same result can be seen at the article, if a doctor would like to be consulted afterwards. The less complex websites seem to represent trustworthy information. In summary it can be said that at first glance the severity of the disease plays an important role, but small differences between the disease and the website version could be determined.

5.7 Impact of User Diversity on Assessment of Websites

In order to investigate to what extent user factors such as age, gender, health status or health competence influence the assessment of health information of varying degrees of severity, a MANOVA was carried out. The health status and the literacy variable were divided into three equal groups. The results show that three significant interactions have been found between gender and assessment, age and health. Women rated the least complex mild disease presentation better than men (\(F(1,\,113)=5,997,\;p=.016\)/Female = 3.39, \(SD=0.93\); Male = 3.23, \(SD=0.92\)). The least complex presentation of the serious disease was rated better by Digital Natives (\(M=3.81,\;SD=1.14\)) than by Digital Immigrants (\(M=3.23,\;SD=0.88\)) and Silver Surfer (\(M=3.13,\;SD=0.77\)) with \(F(2, 113)=4.915,\;p=.009\). Another significant result was found with regard to the health status of the participants. Usually participants with a better state of health (values for mild illness Version: \(M_{bestHealth}=3.57,\;SD=0.7\); \(M_{middleHealth}=3.25,\;SD=0.86\); \(M_{badHealth}=2.75,\;SD=0.85\)) rated the least complex presentation of both diseases better than not so healthy people (\(F_{light disease}(2, 113)=5.382,\;p=.006\)/\(F_{severedisease}(2,\,113)=4.443,\;p=.019\)). Two interactions between gender and literacy as well as age and literacy have been identified. The first interaction was observed in the assessment of the least complex presentation of both diseases (\(F_{lightDisease}(2,\,113)=5,579,\;p=.005\) and \(F_{severeDisease}(2,\,113)=3,854,\;p=.024\)). Women with significantly higher health literacy rated the website version better than men with comparatively low health literacy. The results regarding the mild and severe disease are very similar. Due to the spatial limitations of this article, only one finding according to the mild disease is shown in Fig. 8.

Fig. 8.
figure 8

Interaction of gender and eHealth literacy (low, middle, high) regarding assessment of least complex presentation of light disease. Error bars denote standard error [8].

Full size image
Fig. 9.
figure 9

Interaction of three age-groups (digital native, digital immigrant and silver surfer) and eHealth literacy (low, middle, high) regarding assessment of least complex presentation of light disease. Error bars denote standard error [8].

Full size image

Another interaction can be reported on age and eHealth literacy in relation to the most complex website presentation and mild disease (\(F(4,\,113)=2,655,\;p=.037\)). As can be seen in Fig. 9, the youngest age group with a high eHealth competence value rated the most complex website version better than the middle and older age group.

6 Discussion and Guidelines

Discussion. “In clarity we trust!” sums up one important credibility factor that affects the recipients’ attitude of trusting a health website in our study. We will discuss this aspect as well as further results of our study in this section. We wanted to understand which elements on a website with health-related information convince or rather satisfy people to trust health-related information. Therefore, we conducted a study with two objectives: first of all, we examined factors from websites with health information that trigger credibility. Secondly, we examined to what extent both the media presentation of health websites and the severity of the disease play a role in assessing credibility. Last but not least, we took a look at the nature of eHealth literacy as well as privacy concerns considering general search behavior. To do so, we chose a two-way multi method approach. That way, we could gather more robust results and deal with the complexity of research question in detail. In a first step, focus groups were run, in which we analyzed users’ search behavior when looking for health-related information and collected people’s ideas of factors which make health-related information on websites appear credible.

In a second step, an online questionnaire was sent out in which the results of the focus groups were quantified as well as how digital health information is judged by its recipient and how it varies with the severity of the disease. Findings were analyzed with a diversity focus, thus comparing gender and age groups with respect to media representation, severity of disease, eHealth literacy, and privacy concerns. Insights won from the focus group study show that users, in general, look for a number of factors when deciding whether the information is trustworthy (see Table 5). The factors range from content factors to layout factors. Comprehensibility and sources written in their mother tongue were stressed. User attach importance to understandable and prepared information. Another aspect relates to a clear information structure. A clear structure guides the users and gives them confidence in handling the information. The objectivity of the information is another credibility aspect. Information that conveys a neutral position is more accepted than subjective descriptions of health details. Among the five most important trustworthy elements, source references were also cited. Information about the sources is also considered important. What’s more, users want to know more about the authors’ details. In this respect, our results are consistent with those of Eysenbach and Köhler [13].

When looking to the outcomes in the questionnaire study, again, it was corroborated that users independent of age or gender attach high importance to comprehensibility and clarity as an important credibility element. In our study it was of additional interest to what extent aspects that trigger trustworthiness of information differ for diseases of varying severity. It turns out that the type of factors remains the same, only the order of priorities varies. Comprehensibility remains the most important aspect. While factors such as clarity trigger trustworthiness in the case of a mild disease, objectivity is more important in the case of a more severe disease. Another difference we found was that details about authors are of interest when they inform themselves about a mild disease (in this case hay fever) compared to the publication date, which triggers credibility in the search for a more life-threatening disease. In the latter case, patients have a stronger urge to stay up to date and to not overlook the latest advances in therapy.

One focus of our research was directed to the media presentation of health-related websites. To find out to what extent information on health topics is perceived and evaluated on websites, we have selected three existing electronic health websites and compiled them into fictitious collages without naming the website brand. These developed websites could be distinguished according to their complexity and presentation. In terms of complexity, a website was considered more complex if the information was more detailed and the layout contained more sub-units. It is interesting to note that differences in the perception of credibility are not only due to differences in information complexity. There are no differences in the evaluation when a simple comparison of the means compares their evaluations. Only if user factors or the type of disease are taken into account, differences occur. For instance, gender-sensitive differences could be found when looking at websites with low content and low representation (LowRep) of a mild disease. Women gave a higher rating than men. Moreover, women distinguished themselves with a higher literacy compared to men in regard to assessing a “LowRep” website. It seems that women in general assign a different search and rating pattern than men in this context [23]. Not only gender but also age and health status seem to play an important role when it comes to assessing websites. Younger people seem to be rather satisfied with information of “LowRep” websites than older. The same result turned out for healthy people. However, people with a lower health status seem to have a different point of view regarding the complexity of websites. They rate more complex websites better. This might occur from the fact that people who are less healthy and thus might be more desperate to find solutions to their health problems rather trust well designed websites which are easy to read and understand [23].

We have learned from this study regarding the findings of eHealth literacy and privacy concerns, that people differ in their privacy attitudes when searching for online information. Awareness of data sharing and possible consequences seems to occupy older users more than younger ones. This may be due to the fact that older people are generally not as familiar with Internet applications as are younger people [36]. Thus, older people seem to be more skeptical about Internet activities. This aspect is also reflected in the results in terms of health literacy. Younger participants show a higher competence to evaluate and judge information on the Internet.

Guidelines. Our results show that websites that try to inform the public about health issues must take their information and communication concept into account. In relation to our website evaluation, these results mean that there does not seem to be a “one-size-fits-all” solution when it comes to health information on the Internet. Information providers should be aware of how to design websites for different target groups and possibly consider participatory design methods to determine who needs which information, when, and how. As there are systematic differences in judgments based on health literacy, gender and age, these factors should not be ignored when designing a health-related website. It is important to understand the target group and its health information requirements. Adaptable websites that allow users to seamlessly increase the complexity of a particular disease without hindering clear and easy access to information could provide a solution to such challenges. In this context, the use of a health-conscious recommendation system [37] could be used to determine the information needs of the user depending on the interaction on the website [38]. When other users interact with information in forums or comments, additional, non-verified information enters the stage. In such health-related social media, some users are more active than others [39]. Information and above all meta-information can “drift” through user interaction - especially when algorithms determine the presentation of information (e.g., through evaluation, sympathy). The integration of human oversight into doctor-in-the-loop approaches could be interesting [40].

6.1 Limitations and Future Research

Limitations. As with any empirical study, our results are subject to limitations. The interaction effects studied are suitable for a relatively small subgroups of participants. For example, the older male participants with high health competence are a rather small subgroup of users. This is shown by the size of the larger error bars in the illustrations. Nevertheless, there are effects even with higher error margins. Since our results are consistent with previous studies, further confirmation and transferability of the results would require significantly larger samples or meta-analytical methods to improve evidence.

Since the settings were generated from fictitious websites created by the authors, we cannot be sure that our view of the complexity is shared equally by all users. We have tried to design the websites in such a way that the complexity increases in “equidistant” steps. However, since the texts and images we use come from actual websites, it is not easy to guarantee this. The diseases selected by us (hay fever, breast cancer) have very specific target groups. Men who participated in the study found it difficult for them to put themselves in an environment that required breast cancer therapy. Although men could in fact develop breast cancer, it was strongly regarded as a women’s disease. Furthermore, the question in this study was raised exclusively in healthy participants. At this point, it would be of interest to subject the same research question to a sample with ill participants; especially, on the assumption that personal involvement has an impact on the evaluation of health-related information [16]. As with all scenario-based questionnaires, all results must be taken up with a grain of salt, as the social distortion of desirability could more distort the answers in a more alien environment for the participant.

Future Research. Questions of ethics, data protection [41], and trust naturally play an important role in such solutions we discussed in the guidelines. What information are users willing to share to improve their online experience on health-related websites? Diversity factors play an even greater role here [42]. Therefore, it is necessary to understand the interaction of all user-related factors and the benefits that users see when using such websites. By modelling benefits and privacy, better services or mobile phone applications can be designed with better information quality. The presentation and complexity of these services are adapted to the needs and wishes of users, also taking into account their current usage context. Are you looking for help or just browsing? Should they trust the information they find or should they see a doctor? Either way, the factors that determine the credibility of health information are crucial to help patients, both online on the Internet and offline through a doctor.