Introduction

Melanoma is the most common fatal skin cancer and has a worldwide mortality that exceeds 65,000 people per year, according to the World Health Organization [1]. In 2017, the US incidence of cutaneous melanoma was 87,110 cases and accounted for 5.2% of all new cancer cases with 9730 deaths and was responsible for 1.6% of all cancer deaths [2]. According to the American Academy of Dermatology, the average US cost of melanoma care is $3.3 billion per year [3]. The costs of melanoma care are considerably higher with advanced stages of the disease. Costs for stage I melanoma represent 10% of expenses, while stage IV melanoma accounts for 57% [4]. One of the primary ways to reduce morbidity, mortality, and costs is through primary and secondary disease prevention, which allows for earlier discovery of lesions treatable through local excision [5].

A primary way that people can assess whether they have melanoma and whether to seek care is by educating themselves about what melanoma lesions look like. The most accessible source for this sort of information is the Internet. A 2013 study found that one in three US adults has used the Internet for their own health [6]. The use of the Internet for health information is even more common in cancer patients [7]. For melanoma, a 2005 study reported that 39% of melanoma patients searched the Internet for information; of those, one in three became more anxious after reading what they found [8]. Therefore, guiding patients to appropriate sources of information is vital.

Prior studies have examined the quality of health information on the Internet for cancer. Several studies found that the readability of cancer-related health information (breast, colon, and prostate) tends to be at a collegiate level [9]. A 2016 study reported that melanoma websites demonstrated similar challenges in readability for low-literacy audiences [10]. Studies have also found inaccuracies in Internet websites for cancer-related topics [11]. An earlier study published in 2002 by Bichakjian et al. reported inaccurate information in 14% of 74 evaluated melanoma websites [12]. To our knowledge, no recent studies have directly examined the content and quality of melanoma websites.

Therefore, our study evaluated the content, readability, popularity, and overall quality of consumer-focused online melanoma information. Our goal was to provide physicians with insights about melanoma information on the Internet that patients might encounter and help guide patients to reliable, comprehensive, and understandable educational sources about their condition.

Methods

We conducted a descriptive study of melanoma websites by conducting a standard search of websites that patients or other non-healthcare providers may encounter when searching for melanoma. On October 31, 2017, we searched the four most common search engines: Google (Mountain View, CA), Yahoo! (Sunnyvale, CA), Bing (Microsoft), and Ask.com (Oakland, CA) for melanoma websites. Those search engines were used because they represent more than 96% of all US searches [13]. We typed the term “melanoma” into each of the four search engines and included the first 30 websites in each search engine. Thirty was chosen because prior studies have shown that people infrequently go beyond the 30th website result [14,15,16]. Newly installed browsers Mozilla Firefox browser 54.0.1 and Safari Version 11.0.1 were used to reduce the chance that saved settings or account properties such as “cookies” from previous searches might influence search results.

Starting with 120 websites (30 per search engine), we excluded websites that were duplicates, research articles, websites targeting healthcare professionals, dictionaries, news websites, and websites that had either irrelevant or nonworking links. After the exclusion criteria, 31 websites were included for analysis (Fig. 1). This study was determined to not be human subjects’ research and therefore exempt from review by the Institutional Review Board at the George Washington University.

Fig. 1
figure 1

Melanoma websites according to inclusion/exclusion criteria

Full size image

To evaluate the content, we developed a 22-item checklist for health information of melanoma which we thought would be important to patients. The checklist was adapted according to a prior study conducted on the evaluation of content on melanoma websites on 2002, as well as literature searches, and input from a group of physicians [12]. An initial list was developed from the literature and then presented at research in progress session with more than 20 participating physicians for feedback. After that, feedback was integrated and the items were finalized. The final items evaluated were definition, warning signs, risk factors, assuring signs and symptoms, when to seek medical help, how to diagnose melanoma, types, treatment options, prognosis, prevention, and whether there was an image describing the disease and lesion or not (Table 1).

Table 1 Presence of specific melanoma-specific content items in patient-focused melanoma websites (n = 31)
Full size table

To assess the quality of the websites, several instruments were used. First, we used the DISCERN instrument [17] and used 9 of 16 validated questions to assess the quality of written information on the Internet. Each question assesses a specific quality item, with scoring for each question ranging from 1 to 5. The JAMA benchmark criteria were also used to assess the presence or absence of four items: authorship, attribution (i.e., references), currency (i.e., is there a date?), and disclosures (i.e., any mention of potential conflicts of interest?) [17]. HONcode certification was also used as a measure of quality, which is a voluntary certification that website owners can apply for if they successfully meet the conditions and credibility criteria of HONcode (Health on the Net Foundation) [18].

To assess the readability, we extracted the health information posted on each website which was copied and pasted into automated online readability-assessing tools. The tools that were used are Flesch-Kincaid Reading Ease (FKRE), Flesch-Kincaid Grade Level (FKGL), and Automated Readability Index [19]. FKRE assesses the readability by calculating the number of syllables and words in each sentence and generates a score between 0 and 100, with higher scores indicating easier text to read. FKGL and ARI utilize different methods to generate scores to represent the US grade reading level.

To assess the popularity, we used Alexa.com, which provides information about the popularity of websites by assessing website traffic and ranking on 30 million websites globally. To assess the popularity on social media, we used the website http://shares.webit.pl/ to assess the links of those websites that were shared on social media such as Facebook, Google+, and LinkedIn.

Six physician investigators were trained to extract subjective data from the websites. In this process, a meeting was arranged to coordinate the assessment process and minimize the subjectivity. During this meeting, six websites were chosen randomly, which were assessed separately by each investigator using the content evaluation checklist and the DISCERN instrument. Individual assessment results were among the team discussed, who agreed on ways to best assess the remainder of the websites. Then, the remaining websites were assessed twice by two independent physician evaluators who were blinded to the result of the first assessment and any of the disagreements were resolved via consensus.

Results

Content

The definition of melanoma was provided in 97% of websites. Two of the main risk factors for melanoma which are excessive ultraviolet (UV) exposure and fair skin were mentioned in 90% of websites, and the remainder of risk factors in 71–87% of sites except for weakened immune system and living closer to the equator which were mentioned in 55 and 19%, respectively. The melanoma ABCDE warning signs were mentioned in 97% of sites. Only 19% of sites mentioned reassuring signs that lower the likelihood a particular lesion is melanoma. A total of 84% of websites urged seeking medical care in case of melanoma warning signs. The way physicians diagnose melanoma was described in 87% of sites, and melanoma types were mentioned in only 42%. Regarding treatment and prevention, 90% mentioned treatment options and 74% recommended sun avoidance while 71% recommended the use of hats and protective clothing, and 74% mentioned the use of sunscreen. Given the importance of pictures in assessing the potential for melanoma, only 65% of websites included images of melanoma lesions (Table 1).

Quality

Health On Net Code Certification and Journal of the American Medical Association (JAMA) Benchmark

Only 8 (26%) of the 31 assessed websites were HONcode certified. Regarding the JAMA benchmark criteria, 13 (41%) mentioned authorship, and 13 (41%) websites mentioned sources and references. Nineteen (61%) websites disclosed their ownership, sponsorship, adverting policies, or conflict of interest, and 17 (54%) websites mentioned the date of post or date of update.

DISCERN

A total of nine questions from DISCERN were answered and median values as well as interquartile ranges (IQR) were calculated. The highest three ranked DISCERN questions were “Is it relevant?” followed by “Does it achieve its aims?” and “Are the aims clear?” Each of these questions had a median score of 5 (IQR 4–5). The three lowest ranked questions were “Does it refer to areas of uncertainty?” (median 2, IQR 3–4), “Is it clear what sources of information were used to compile the publication (other than the author or producer)?” (median 1, IQR 1–5), and “Is it clear when the information used or reported in the publication was produced?” (median 3, IQR 1–3) (Fig. 2).

Fig. 2
figure 2

Questions on melanoma websites rated on nine questions for DISCERN. The red line is the median and the gray bar is the interquartile range for each question

Full size image

Readability

The median number per website of words was 2992 (mean of 2574, standard deviation 954, and range of 508–3529). A total 12 (40%) websites were readable by grades 8–9, and 12 (40%) were readable by grades 10–12. The remaining 6 (20%) were readable by grades 13–14. One website could not be analyzed because of a low word count (< 100).

Popularity

Using the Alexa website, we present the top 5 ranked websites based on the number of visits and page views in the USA and globally (Table 2). We also present the top 5 shared websites on social media: Facebook, Pinterest, and LinkedIn (Table 3). These websites demonstrate variable readability and quality (Table 4).

Table 2 Melanoma websites’ global and US ranks
Full size table
Table 3 Melanoma websites’ popularity on social media
Full size table
Table 4 Quality and readability of most popular websites
Full size table

Discussion

In this study, we evaluated the quality and content of melanoma, the most common fatal skin cancer [20]. Public awareness of melanoma is important because early treatment is curative. International screening programs focus on identifying lesions early so they can be treated before the development of metastatic disease. One of the major ways of reaching the public is through websites on the Internet, which is a primary source for health information [21]. In our review of melanoma websites intended for patients, we found many that provide high-quality information for patients. However, we also found that content and quality vary greatly and that many websites are difficult to read, particularly for lower-literacy population. Many websites also lack references and are not regularly updated.

However, compared to a 2002 study, we found that melanoma websites have improved considerably. Today’s websites have more content, notably improved definitions, risk factors, treatments, and information on prevention [12]. In particular, the information is primarily focused on the diagnosis of melanoma, and people can use melanoma pictures from the web to compare to their skin lesion in more than two-thirds of websites. Additionally, the warning ABCDE signs can be useful adjuncts to pictures [22]. People may also search for reassuring signs which lower the likelihood that the lesion is melanoma; however, this information was provided in less than a third of the websites. Once diagnosed, patients may use websites to understand treatment options and other diagnostic tests, which was very commonly mentioned in nearly all melanoma websites.

However, one problem that has remained with many websites is the quality of the information itself and how it is referenced and updated. Specifically, the DISCERN items tended to be rated low, specifically the date of publication which was absent in one in three websites. Also, more than half of the websites did not contain references, which are important given the evolving treatment options in melanoma. While nearly all melanomas require surgical excision, medical treatment is constantly changing. New treatments have emerged over the past few decades, such as IL-2 and pegylated interferons, approved by the FDA in 1998 and 2011 respectively [23]. Authorship information was missing in nearly half of the sites which makes it unclear to readers whether the writer is an authority in melanoma care. High variability in website quality has also been shown in previous articles on different types of cancer [24, 25]. Several reasons may also underlie this. First, there is no website standardization, and there is diversity in the focus and objectives of websites (educational vs. organization vs. treatment center). Also, there are variations among authors in expertise and focus [26]. For example, MD Anderson’s website is mostly focused on treatment while Aimatmelanoma’s website takes a primary diagnostic approach. In addition, only a minority had HONcode certification—a voluntary certification for websites with health information; however, in recent years, HONcode has started charging which may deter websites from using it.

The readability of websites also varied, with almost two-third of the websites being difficult to read for people with less than an 8th-grade or 9th-grade education. This confirms the findings from another recent study on melanoma websites [10]. Current recommendations from the American Medical Association suggest that Internet health information should be written at a 4th- to 6th-grade reading level [26]. This finding is consistent with prior studies that demonstrated that much of the health information on the Internet is not readable for low-literacy populations [27,28,29]. When it comes to melanoma in particular, the complexity of oncologic terminology contributes to higher readability levels [28]. However, several websites were easier to read and also had good quality such as American Society of Clinical Oncology’s website [30]. This suggests that good information on melanoma can be communicated in an easy-to-read manner, and websites should strive to be readable for everyone.

For melanoma, Wikipedia was the most popular website globally and nationally, which has been a concern given its open-source nature where anyone can update and edit content. However, we found Wikipedia information to be clear and accurate. However, it was not accessible to readers below a reading grade of 12th grade. We also found that the AIM at Melanoma Foundation website had high-quality information but also was fairly difficult to read for low-literacy readers (10th grade). Some popular websites, including the American Melanoma Foundation website, were very popular yet ranked low in quality, suggesting that greater popularity does not necessarily mean greater information quality.

There are several limitations to this study. First, only the first 30 websites were included for assessment. It is possible that results would have been different had we reviewed more websites. Among the physicians who assessed the websites, there are no dermatologists or oncologists. However, evaluating website content does not necessarily require specialist training. It is also possible there was subjectivity in assessing content as well as applying tools; however, we used multiple methods to ensure that consensus was reached. Finally, the content, as well as the popularity of the websites is constantly changing. Therefore, we cannot generalize results to any particular time beyond the date the information was extracted. Our study also did not evaluate specific videos or pictures embedded within or linked in the website. Lastly, only websites in English were evaluated.

In conclusion, we found that there are many good-quality websites for patients with melanoma. However, a large number of popular websites had inconsistent and variable content, and most websites require high educational levels. In addition, the quality and content of melanoma websites has improved considerably since the last time they were reviewed in 2002, more than 15 years ago.