Introduction

Low health literacy of healthy people entails a number of risks and problems. Among others, screening (e.g., mammography) and preventions (e.g., influenza vaccination) are used less often. For patients, low health literacy is associated with more hospitalizations, higher frequency of emergency room visits, more false prescriptions and less adherence. Accordingly, low health literacy puts persons at risk of a poorer general health status. Furthermore, these persons face bigger barriers to access appropriate medical care. Moreover, patients with low health literacy not only have less access to health information but they also get less involved in the decision-making process. In contrast, patients with higher literacy get more active and have less decision conflicts, higher treatment satisfaction, higher adherence, improved health behaviors, and a better global health status. As a result, higher health literacy improves therapeutic safety. From the perspective of a health care system, higher health literacy is associated with a reduction in unnecessary medical procedures and less consumption of resources [1,2,3].

For cancer patients, poor health literacy may reduce effectiveness of communication with health care professionals thus causing misunderstandings on status of disease and treatments. As a consequence, anxiety may increase and adherence may drop. False or unnecessary interventions may reduce prognosis as well as quality of life [4].

For these reasons, improving health literacy is a major concern of the German National Cancer Plan and similar initiatives in other western countries. This includes, inter alia, a barrier-free access to target-group-oriented and quality-assured information. As a result, the patient’s competence is increased, his autonomy strengthened, and shared decision-making will be promoted [5].

Reducing the potential hazards caused by inadequate health information can only be ensured through access to information of high quality. The pros and cons of any medical procedure should be presented in detail using scientific evidence and reliable sources [6, 7]. For a long time, the physician was assumed to be the most important source of information for patients. A recent study by the Bertelsmann Foundation showed, however, that the physician is only in the fourth position. Younger patients (up to 60 years) und those with a higher education use books or the Internet as main source of information. Meanwhile, older patients and those with a low level of education prefer free brochures and magazines [6].

In 2005, a German expert group set up scientific-based recommendations on high-quality evidence-based patient information, which by now are updated and available on the Internet. The key points of the information are validity, reliability, completeness, and relevance [7].

Furthermore, several organizations have developed instruments to assess the quality of health information in general or specific for health information on the Internet:

  • For health information on the Internet, a frequently used rating catalog is the HONcode. It represents an ethical code of conduct for publishing medical information on the Internet. Once the eight basic principles from the code of conduct are obeyed, the HONcode seal may be attached to the web page [8].

  • Another option is the examination of 10 criteria of transparency in order to obtain the seal for certified health information (afgis seal), which is valid for 1 year [9].

  • In Germany, the DISCERN instrument is frequently used during the production of patient information. The DISCERN instrument is a standardized questionnaire for developers and users of written health information, which was developed in a multi-phase process in 1998 [10].

  • Furthermore, health information can be tested by the Check-in tool, which was developed to assess the quality of health information [11].

Not only these publications, criteria, and seals, the quality of health information on the Internet is also highly heterogeneous. Websites with low-quality information are often more visible on the Internet than are websites with high-quality information [12].

From the abovementioned patient information-seeking behavior, one might conclude that booklets which can be downloaded from the Internet for free may be an attractive source of information for patients with cancer especially for those with lower health literacy who access the Internet yet prefer to read a booklet.

In order to learn more on the currently available booklets, we decided to evaluate the quality of information provided in free-access booklets available on German websites.

Material and Methods

Assessment Criteria

In 2014, our working group developed a merged instrument for the assessment of web-based information for lay-people. We compared and compiled both the content and formal criteria by matching different requirement catalogs:

The first step was to compare the HONCode principles [8] with the DISCERN instrument [10]. Content and formal criteria were differentiated. Next, the criteria for the quality assessment of health information [13] were integrated and then the criteria for evidence-based patient information [7]. As a last step, we added the transparency criteria of the afgis collaboration [9].

Finally, from this extensive collection of requirements, an assessment catalog with a total of 16 items within 4 categories (quality of the publication, quality of information, quality of information representation, and transparency) was created (see [12]).

This instrument was first tested for understandability and reproducibility on a small set of websites.

Selection of Patient Information Booklets

All of the patient information booklets had to be available for free on the Internet. Booklets that were only available as print version from an institution or interactive health information were not counted. The six most common tumor types in Germany for women and men respectively were selected [14]. Thus, nine different tumor types were taken into consideration: breast cancer, prostate cancer, lung cancer, colorectal cancer, endometrial cancer, bladder cancer, melanoma, pancreatic cancer, and oropharyngeal cancer.

From a former analysis of websites supplying information for cancer patients, we were able to identify the websites with booklets for cancer patients [12] (the procedure was described in detail in [15]).

Evaluation

The evaluation of the booklets was carried out by three independent experts: two scientists with experience in evidence-based medicine and creating patient information, as well as a medical professional with main focus on patient information. The evaluation of the 16 items was done using a three-point Likert scale (0 = criterion is not met, 1 = it is partially met, 2 = it is completely fulfilled). For every item, the points from the three experts were added and the sum was divided by 3, thus generating in a mean value. Additionally, we calculated arithmetic mean values for each item, for the 4 categories (quality of publication, quality of the information, quality of information presentation and transparency) and for the whole booklets.

In the overall assessment, a value of 1 would be associated with a high risk that essential elements would be missing. Achieving an overall mean of 2 and therefore completely fulfilling all criteria would be desirable but a disproportionate demand. Therefore, at least a value of 1.8 (=80%) in the overall assessment should be achieved for a recommendable source.

In order to find out whether there are significant differences with respect to the underlying disease, in a final step, we summarized the mean values of the criteria resulting in a mean score for each criterion and tumor type.

Each expert was blinded.

Statistical Methods

We used SPPS version 23 to evaluate frequencies and correlations.

Ethical Vote

According to the rules of the Ethical Committee of the J.W. Goethe University, Frankfurt/Main, no ethical vote was necessary for this study.

Results

Characterization of Patient Booklets

The acquisition of booklets was carried out between February 14 and 21, 2015. A total of 52 different patient booklets were downloaded and assessed. In sum, 26 booklets (50.0%) were available from the different Cancer Societies, 10 (19.2%) from the German Cancer Aid, 1 (1.9%) from the Cancer Information Service, 7 (13.5%) from the websites of the pharmaceutical industry, 5 (9.6%) from the websites of different statutory health insurance companies, and 3 booklets (5.8%) from the websites of the self-help organizations. Most booklets referred to the three most prominent types of cancer: breast cancer (14; 26.9%) followed by prostate cancer (8; 15.4%) and colorectal cancer (7; 13.5%).

Evaluation of the Booklets

None of the booklets scored a mean value of 2.0 as none met all criteria. A mean value of 1.6 was achieved by only 14 of the 52 booklets (27%). In detail, only 11 booklets from different cancer societies (21%), 1 booklet from the German Cancer Aid (1.9%), 1 booklet from self-help organizations (1.9%), and 1 booklet from a statutory health insurance company (1.9%) could be rated as a recommendable source of information.

The first item was evaluated at the highest value with an average of 1.92 (“The aim of the publication is clearly described”). Most items were rated between 1 and 1.5. The lowest average rating was 0.49 for item 14 (transparency criterion: “Are the sources sufficiently proven?”). A detailed presentation of the results can be found in Tables 1 and 3.

Table 1 Item rating by type of cancer
Full size table

In sum, the quality of the publication (Item 1 to 4) was evaluated with an average value of 1.67; this criterion was the one with the highest rating. Quality of the information (items 5 to 10) was the second best with a mean value of 1.45. The quality of information presentation (items 11 and 12) showed a similar rating (1.39). The transparency criteria (items 13 to 16) were evaluated as the lowest with an average of 1.07. A detailed presentation of the results can be found in Tables 2 and 4.

Table 2 Category rating by type of cancer
Full size table

With respect to the single items, there were several criteria with a rating below 1.5. From these data, support of shared decision making ranked the first with 1.47 followed by qualification of the authors, appropriateness or understandability of the illustrations, no statement without reliable information, and completeness of information (all 1.31 to 1.32). Lower scores were given to scientific proof of information and transparent origin of information (0.96 to 0.97) and the lowest to substantiation of the sources (0.49). See Tables 1 and 3 for further details.

Table 3 Item rating by origin
Full size table

Compared to the mean value of booklets of all cancer types, booklets for breast cancer and endometrial cancer ranked the worst in almost all items and in all categories. In contrast, booklets of colorectal cancer and prostate cancer as two other major cancer entities were often rated better than average (8 of 14 items each, 3 and 4 categories resp.). See Tables 1 and 2 for further details.

With respect to the organizations responsible for the booklets, booklets from the German Cancer Aid, the Cancer Societies, and the Cancer Information Service achieved better results in 10 of the 16 items and three categories compared to the booklets from self-help groups, pharmaceutical companies, and statutory health insurance companies. In the remaining 6 items (3, 4, 9, 10, 11, and 12), booklets of the Cancer Information Service showed the lowest results. See Tables 3 and 4 for further details.

Table 4 Category rating by origin
Full size table

The ratings for three of the quality categories were more or less homogeneous for the booklets addressing different types of cancer (quality of publication ranged 1.54 to 1.85, quality of information ranged 1.37 to 1.65, quality of information presentation ranged 1.20 to 1.51), and those for transparency had a broader range (0.88 to 1.35). Furthermore, for the single items, the ratings were more or less homogeneous with respect to the different tumor types. Yet, three items showed a broader range of ratings: appropriateness/understandability of the illustrations ranged 1.08 to 1.67, transparency of source of information ranged 0.67 to 1.25, and substantiation of the sources 0.17 to 1.08. See Tables 1 and 2 for further details.

In the evaluation addressing the origin, the category of quality of information presentation was evaluated best in the booklets of the statutory health insurance companies. The highest value in the quality of publication category was achieved by the booklets of the German Cancer Aid. The quality of information presentation was best met by the booklets of the German Cancer Aid and the Cancer Information Service. The criteria of transparency were completely fulfilled (2.00) by the booklet of the Cancer Information Service. On the other hand, the booklet of the Cancer Information Service showed the lowest score in the quality of information presentation. See Table 4 for further details.

Discussion

Overall, no booklet achieved a maximum rating of 2.0. Yet, booklets from the German Cancer Aid, the Cancer Societies, and the Cancer Information Service were mostly rated higher compared to the booklets from self-help groups, pharmaceutical companies, or statutory health insurance companies.

One reason for this might be because of the process of creating the booklets. For the professional societies, development and drafting are done by experts in their respective fields, mostly supported by specialists for the production of health information. Additionally, cooperation with external scientific consultants is often built. Furthermore, some of these booklets are adaptations for lay people of the national consensus cancer guidelines that are produced in a structured process with quality control, since high-quality information seems to be expectable.

Financial resources may not be the sole explication for the differences. In fact, the German Cancer Aid, the Cancer Information Service, and statutory health insurance companies are supposed to have adequate resources. Yet, the budget of the regional cancer societies is heterogeneous. The low score of the booklets of the self-help groups may be due to the missing resources with respect to access to experts as well as financial resources. On the other side, many experts voluntarily offer help to patient groups. The low rating may also be due to a different ranking of quality criteria of patient information by patient advocates. It might be that for patients, the subjective point of view instead of scientific evidence is important [16]. Another point could be that patient advocates appreciate disclosures and other formal criteria in case of publications from other institutions but do not think them important for a publication from self-help groups.

The results of our investigation are consistent with other studies in which comparable results or deficiencies were detected. Walling and colleagues indicated in the evaluation of a total of 29 prostate booklets, from the 85 predefined subcategories, only 22 of the high-rating subcategories were achieved [17]. In this analysis, the presented information was either incomplete or inaccurate. In another study on prostate cancer booklets, Weintraub et al. showed that 25 of the 29 examined booklets (86%) were not adequate to support a shared decision [18]. In a further tumor-specific analysis by Rees, Ford, and Sheard of 31 booklets on prostate cancer, the majority was rated as moderate or not useful at all [19]. A similar result was reported by Smith and colleagues in a study of 31 booklets on colorectal cancer [20]. Ninety percent (28 of 31 booklets) were rated as insufficient and even 4 booklets (13%) were rated as unsuitable [20]. Slightly better results could be observed by the working group led by Ryan who examined 97 different information booklets dealing with oncological topics [21]. In this study, most booklets were rated as outstanding (59.3%) or adequate (14.8%) and only a quarter was insufficient (25.9%) [21].

It was found that the linguistic adaptation to the lay audience (item 9 and 11) was rated at its lowest in the booklet of the Cancer Information Service. Accordingly, texts from the booklet of the Cancer Information Service were evaluated by a Support Vector Machine which classified the level of language of lay texts as too challenging for laymen [15].

Concerning scientific criteria (2, 6, 7, 8, 14 and 15), booklets from health insurance companies, pharmaceutical industry, and self-help groups showed lower values than the average. Referencing to scientific sources was only used sparsely in many booklets. This was true especially for booklets of the German Cancer Aid. In contrast, booklets from self-help groups complied best with this criterion. The high compliance of the self-help groups with referencing might point to their acknowledgement of patients’ need of highly qualified information [22].

Booklets may help the patient in preparation and post-processing of discussions with the physician and should encourage shared decision-making. Booklets about lung, breast, and endometrial cancer and melanoma provide less support than others. Especially for breast cancer, this is astonishing as in many situations, different treatment options exist and a great amount of research on communication has been done with this group of cancer patients. Likewise, these booklets less often offer decision aids. Most importantly, booklets from a wide range of providers do not support (shared) decision-making substantially. This not only is true for booklets from the pharmaceutical industry but also for self-help organizations, statutory health insurance, and the Cancer Information Service. In this case, the booklet of the Cancer Information Service received the lowest rating. While the pharmaceutical industry might be interested in influencing the patient in a certain direction, for all others, this finding is surprising. It might be that the skills to further (shared) decision-making are lacking. Yet, this is not to be expected from a professional organization focusing on the information of patients as the Cancer Information Service. One explanation might be that texts are complicated. Thus, uncertainty of patients might increase rather than decision-making be encouraged if the authors want to describe pros and cons of different options in detail and without influencing the patient [23].

Regarding transparency criteria, booklets from the pharmaceutical industry, statutory health insurance, and self-help organizations got lower results than the average. In contrast, the booklet of the Cancer Information Service fulfilled these criteria completely. For self-help organizations, the explanation might be similar as above—if they mostly prepare these booklets for their members, these may be supposed to know the organization and its structures. For physicians who pass booklets from the industry to their patients, this result, in addition to other findings, is important. Without thoroughly reading and proofing, no booklet should be passed to the patients who may not be able to detect influences from the provider. Physicians may not rely on the quality of health information as internal mechanisms on the side of producers may not be sufficient despite a general availability of the criteria. Accordingly, physicians should scrutinize any booklet they recommend or pass to patients. Moreover, high-quality information must not only include the best available external evidence but additionally be readable or understandable [6]. Therefore, physicians should not only assess content and formal criteria but also examine the readability or better understandability. Furthermore, also health insurance companies might have certain interests to influence the patient. Together with the fact that their booklets also are less supportive concerning decision-making, physicians should be attentive of any exertion of influence.

Limitations

The search of patient booklets was carried out through careful perusal of the web pages. Nevertheless, it is possible that not all available booklets could be downloaded and evaluated as booklets may have been overlooked. Another methodological limitation is the fact that the brochure had to be downloaded free from the internet. Hence, booklets that were not available for free on the Internet, but could be sent free by conventional mail, were not taken into account.

Furthermore, a bias was possible. A blinding could not be guaranteed due to the evaluation criteria of item 13 (“Is it clear from whom the information originates?”) and item 16 (“Is there information on the financing of the publication?”).

Due to the small sample size, no analytical statistics could be applied either in the consideration according to the origin or in the tumor-specific consideration of the booklets. However, the key message of this study was not affected.

Conclusions

In summary, we were able to demonstrate formal and content deficits in German booklets for cancer patients. This is in line with studies from other countries. The applied criteria cover all internationally required content and formal criteria. While enhancing health literacy of laymen is a major concern in the western world, the majority of written information is not suitable for this goal. Furthermore, despite several decades of research and discussion on shared decision-making, most booklets are not supporting shared decision-making. Physicians should scrutinize any booklet they recommend to patients not only for its quality of health information but also its comprehensible quality, while organizations as the German Cancer Society might regularly assess, rate, and rank all available booklets manually or with the aid of IT programs. At the end, a multimodal test panel as part of an obligatory external evaluation could help to improve the quality of health information.