Introduction

Esophageal cancer is the seventh most common cancer globally with nearly 600,000 patients diagnosed each year and is prevalent in both developing and developed countries [3, 15]. The multidisciplinary treatment of esophageal carcinoma may include options such as open surgery, minimally invasive techniques, endoscopic resection, chemoradiation (either adjuvant or neoadjuvant), and palliative care, which may confer considerably varying clinical outcomes and quality of life of patients [14, 23]. A better understanding of the disease and its care pathways may help patients to make better informed decisions regarding their treatment and help patients navigate through complex cancer care pathways.

The Internet is undoubtedly the largest source of information worldwide and is accessed by more than half of the world’s population [17]. Studies have shown that approximately 60% of adults had used health information available online on a range of health topics. Furthermore, 35% of the adults have been shown to go online to learn about medical conditions they or someone else known to them might have [12]. The quality of the available information online has become a matter of concern with the widespread use of the Internet in accessing health information [11, 24]. There is also a concern due to the lack of standardization and regulations to ensure accuracy of the information [11]. This has resulted in a wide discrepancy in the quality of information reaching the patient [11]. On many occasions, the information available on the Internet had been found to be of substandard quality [1, 16, 25]. Despite these concerns with the quality of the information, a majority of patients consider these websites to be reliable, which may negatively impact patients’ decision making [1].

Among the general public, most Internet users possess an average level of education and reading skills [19]. Medical language and concepts can be very complex for the public and it is essential that these websites have health information that is easily understandable regardless of age, educational background, or reading level [19]. According to the guidelines published by the National Institute of Health, the language used should be below the seventh grade level to ensure easy readability [19]. However, the information available online does not seem to comply with this guidance as many studies have shown the complexity of language to be beyond what an average person can understand [20].

A previous study by Burke et al that evaluated the information available on the Internet regarding esophageal carcinoma has shown that there were quality issues with most of the information. Although there were high-quality websites providing patient information about esophageal cancer, the mean DISCERN score was 51.5 indicating moderate quality. Certain sites had a minimum score of 15 and none of the websites had reached a high score of 80 [5]. However, limited search strategies had been used in these studies, and furthermore, the readability, usability, and reliability have not been assessed. We conducted this study with the aim of analyzing the quality, usability, readability, and reliability of patient information websites on esophageal cancer available through popular search engines using a wide search strategy and validated tools.

Methods

Data Sources

The three most popular search engines (i.e., Google, Yahoo!, and Bing) were searched for patient information on esophageal cancer. The primary search for the websites was performed in September 2019 and was repeated in March 2020 in search of newer websites using Google chrome browser. These search engines rank their search results based on factors such as the popularity of a given link and the number of hyperlinks pointing to it from other webpages. Furthermore, these search engines may also produce search results both relevant and important by combining page rank along with text matching techniques [4]. Therefore, the search results may vary over time. To overcome this limitation, a repeat search was used to identify websites that were missed in the previous search. The search was performed in English using the keywords esophageal cancer, esophageal tumor, esophageal tumor, esophageal malignancy, esophageal cancer, esophageal tumor, esophageal tumor, and esophageal malignancy. All searches were performed in incognito mode to minimize bias created from previous searches. The search was performed using default settings avoiding advanced settings or plugins to enhance the output.

Over a million search results were generated by each search engine for a single keyword (average approximately 4.5 million per keyword per search engine) (Table 2). The likelihood for a patient to view a particular article that comes up in a search is greatly influenced by the order of appearance on these popular search engines. Research had shown that most of the users only visit the top ten search results produced by the search engine with less than 5% accessing websites beyond the first ten websites [10]. However, the first 50 search results were analyzed from each keyword to avoid missing important websites. For each website information such as the URL, website title, year of publication, and the last date of update were recorded.

The inclusion criteria considered in the selection of websites were the lack of password requirement, free access, available in English language, and providing targeted patient information on esophageal cancer. Blogs, duplicate websites, and news articles, as well as sites that contained advertisements and discussions, were excluded. Sites with professional articles and guidelines for clinicians were also excluded as they were beyond the scope of the study.

The websites were selected and assessed by two independent investigators. Prior to the assessment, the scoring process was piloted to ensure the proper use of instruments by the investigators. The credibility of the specific information on esophageal cancer including clinical presentation, methods of investigation, modalities of treatment, complications of treatment, support groups, and quality of life were assessed. The accuracy of the content was assessed using the European Society for Medical Oncology (ESMO) clinical practice guidelines for esophageal cancer as the gold standard [22]. The specific details on esophageal cancer that were assessed by the websites were given an all or none score based on the guideline.

Readability Scores

The ease of comprehending and understanding of a given text is referred to as its readability. Flesch Reading Ease Score (FRES) which is the most widely recognized and the most commonly used score to assess the readability was used to assess the readability in this study [21]. The FRES score determines the level of education a reader must possess to be able to read and understand a text. The formulae used generate a score between 0 and 100. Higher scores indicate greater ease of reading [21].

LIDA Tool

The LIDA instrument was used (The LIDA Instrument, Version 1.2, 2007, Minervation Ltd., Oxford, UK) for the analysis of the content and the design of health-related information of the selected websites. The LIDA instrument assesses quality, accessibility, usability, accuracy, and reliability [18]. The usability and reliability domains in the LIDA score were used in this study [18]. The usability depends on the content’s understandability when displayed on the website. The clarity of presentation, functionality with intuitive browsing, inbuilt search facilities, consistency of the design, and engagability of the users are the components included for usability. The components of reliability include regular updates, accurate methodology for content production and output, and declaration of conflicts of interests. The responses were given grades from 0 to 3; 0 being never and 3 being always. Scores more than 90 were considered “high,” scores between 50 and 90 were considered “moderate,” and scores less than 50 were considered “low” [18].

DISCERN Instrument

The DISCERN questionnaire was used to assess the quality of the published material. It is composed of two groups and a total of 16 questions. The reliability is assessed by the first eight, treatment information by the next seven and the question 16 assesses the overall quality of information displayed on the website related to treatment, sources of information, additional support, and benefits and side effects of treatment. Scores range from 1–5 for each question which gives a maximum possible DISCERN score of 80. Each question has a set of guidelines and hints for users to understand the important concepts [6, 9, 13].

Piloting the Study

The study instruments were introduced to the two investigators before scoring the websites. Instructions were given on components of the instruments and the methods of scoring. The sources leading to disagreement and difficulties were identified and clarified. The two investigators independently evaluated ten separate patient-related websites on different topics, and any disagreements between investigators were resolved via consensus using a third investigator. The above process was repeated until an agreement of more than 90% was reached between the two investigators.

Assessment of Websites and Data Collection

A process similar to piloting was followed for data collection and scoring. The websites fulfilling the inclusion criteria were independently analyzed and scored by the two investigators. A common agreement was reached on the individual score for each DISCERN question for each website. In case of ambiguity, an independent third investigator was involved as the tiebreaker.

Website Certification

The credibility and reliability of online health-related websites are standardized by online certification sites. The Health on the Net (HON) code is the oldest out of the available quality evaluation tools created by the Health on the Net Foundation in 1995. It comprises of an 8-point code of conduct. The HON code–approved websites will display a badge indicating approval. The HON foundation annually reviews the HON certification of the certified websites. Greater than 7,000 websites from 102 countries are HON code certified [11].

All websites were classified as governmental or non-governmental based on the institution responsible for the maintenance. The websites maintained by the country’s public health authority were considered governmental and sites managed by private institutions, non-governmental organizations, or voluntary cancer institutions independent from the government were classified as non-governmental.

Data Analysis

The SPSS (version 20) software was used to analyze the data and associations were determined using nonparametric tests. A p value of < 0.05 was considered statistically significant.

Results

Out of the top 1,200 websites obtained from the three search engines, using all keywords, a total of 1,092 websites were excluded due to duplication and the target readers being healthcare professionals. The remaining 108 websites were included in the analysis. A great majority of these sites (92.6%, n = 100) were owned by non-governmental, not-for-profit organizations maintained with the aid of donations and fundraising campaigns. Only 30 websites (27.8%) were found to have a current HON code certification.

The median FRES score of the included websites was 48.25 (range: 15.6–70.1) which is equivalent to college-level readability. Only one website (0.9%) scored more than 70, which is the accepted standard of readability according to the National Institute of Health guidelines. The median LIDA usability and reliability scores were 46.5 (range: 22–54) and 39.0 (range: 10–51), respectively. The overall median LIDA score was 85.0 (range: 36–103), whereas the median DISCERN score was 50.5 (range: 23–79). Figures 1 and 2 illustrate the performance of websites on individual criteria assessed by the DISCERN tool. A significant positive association was observed between the DISCERN score and LIDA usability, reliability, and overall LIDA score (p < 0.001). In addition, the DISCERN score had a significant positive correlation with the HON code certification (p = 0.01) (Table 1). The characteristics of the top 10 websites based on the DISCERN tool are given in Table 3.

Fig 1
figure 1

Website performance on individual criteria assessed by the DISCERN tool (questions 1–8)

Full size image
Fig 2
figure 2

Website performance on individual criteria assessed by the DISCERN tool (questions 9–16)

Full size image
Table 1 Correlation between DISCERN scores and other factors
Full size table

The accuracy, reliability, currency, and content production were also assessed. With respect to the currency of information, only a small proportion of websites (50%, n = 54) have stated the last date of update or the frequency of update (32.4%, n = 35). Similarly, the credibility criteria such as acknowledging the author and citing sources were achieved only by 54.6% (n = 59) and 29.6% (n = 32) of the sites, respectively.

The specific information provided regarding esophageal cancer is shown in Fig. 3. Reliable clinical information was provided in 98.1% (n = 106) of the websites, accurately mentioning the symptoms, while 88.9% (n = 96) had detailed accounts of investigations and treatment options. However, the information on the consequences of “no treatment” and effects of different treatment options on quality of life was given in only 19 (17.6%) and 48 (44.4%) websites, respectively. Information on the benefits (27.8%, n = 30) and risks (30.5%, n = 33) associated with different treatment methods were included in less than a third of the websites.

Most of the sources referred to surgery as the main modality of treatment. However, only 48.1% (n = 52) mentioned adequately detailed accounts of the different surgical procedures. Only 47.2% (n = 51) of websites had integrated non-textual media, to explain the procedure. Endoscopic mucosal resection (EMT) that is used to remove early-stage tumors was described in 65.7% (n = 71) of the websites. Details on chemoradiotherapy, palliative care, and prognosis were stated by 83.3% (n = 90), 53.7% (n = 58), and 63% (n = 68) of the websites, respectively. Only 44.4% of the sites have given details about clinical trials and advocated patients to take part.

Discussion

The recent advances in information technology, along with the ease of access and availability, have led to an unprecedented number of online users of consumer health-related information. The present study has demonstrated considerable issues with many of the quality parameters of patient directed online information on esophageal cancer.

A clear indication of the content production method and names of contributors help improve the credibility of information, and indicating the date of the publication ensures the patients are aware of the currency of information included in a patient information website. Despite that, our study revealed that over half of the websites did not conform to these. This was further confirmed by the low LIDA reliability scores obtained by a majority of the websites. In addition, the median readability level of the websites was found to be equivalent to college-level readability although the recommendation is to have it below seventh grade level. This not only leads to limited understanding of the information by a majority of the users, but risks misinterpretations with potentially harmful effects. On the other hand, the top 10 scoring websites with the DISCERN had a median score of 77 out of 80 indicating that it is feasible to publish good quality and reliable information on a website.

Writing online consumer health information is a difficult task, especially when it is done by a highly educated expert in the field who is more familiar with complex medical jargon than simple lay language. The readability improves when using short sentences, using active voice in writing and using examples in explanations. Furthermore, the use of 12-point or larger font sizes and the use of illustrations as appropriate had also shown to improve the readability [8]. Incorporation of non-textual media including videos and diagrams improves not only the understanding but also patient satisfaction and recall [2]. Despite that, use of these simple measures including non-textual media usage was observed in less than half of the websites.

In relation to content, a patient information website ideally should cover all facets of the index condition including diagnosis, treatment options, outcomes (with different treatment options including with no treatment), quality of life, and palliative care [7]. Although tools including LIDA and DISCERN give a general evaluation of reliability, they do not evaluate the comprehensiveness or accuracy of the information. This remains a limitation in using these tools to assess quality of patient education websites.

As more and more people access online health information, it is important to implement protocols to certify websites based on their measurements of quality such as accuracy, reliability, and readability which may facilitate consumers to find the right information by accessing certified websites. Until such certifications are widely implemented, patient information sites can opt to show the scores of reliability, quality, and readability on their webpages to guide the readers. Furthermore, national cancer services should take the lead and play a more active role in developing good quality information for patients with all cancers including esophageal cancer. Such websites have the additional advantage of being able to display information that suits local context which include available investigations, treatment options, outcomes, and available cancer care services and how to access them in a more socially and culturally acceptable manner.

There were several limitations in this study. Although we used the most frequently used search engines worldwide with default settings, these are subjected to variable results depending on the geographical location. Furthermore, these search engines use various complex algorithms to rank results which change regularly. Hence, the exact results of our study may not be reproducible, although we believe the general results of the study holds valid. Furthermore, the scoring tools which were used to assess quality of information analyze only a few selected parameters and hence likely include limitations in the assessment of quality. However, as previous studies have shown, these tools are useful especially when it comes to comparisons among websites and to assess changes over time.

Conclusion

The Internet is a crucial resource for dissemination of health information in the modern world. However, the quality of majority of the websites providing patient education, with regards to readability, usability, and reliability scores used in this study, was suboptimal. With the enormous volume of online health information, better informed decisions on treatment, follow-up, and prognosis may be achieved with access to good quality information online. A multifaceted approach to improve the quality of patient information websites is essential to ensure good quality information reach patients seeking health information online.