INTRODUCTION

Patient Education Materials

The 2008 Health Tracking Physician Survey from the Center for Studying Health System Change (HSC) reported that 75% of physicians routinely hand out patient education materials (PEMs).1 PEMs improve patient self-efficacy, thus supporting the growing trend towards disease self-management.2–4 Additionally, they are potentially effective at improving patient comprehension and influencing health behaviors, especially if they are written at appropriate reading levels for patients.5–7 The Joint Commission8 states that PEMs should be written at or below a 5th grade reading level, and encourages hospitals to use readability tests to revise written materials in order to address the health literacy needs of all patients.

Health literacy is a critical issue. The average US resident reads at an 8th grade level,9 and the average Medicare beneficiary reads at a 5th grade level.10 These statistics have implications for patients, including their ability to understand common medical terms. In a study of 249 adults at a metropolitan Emergency Department, investigators found that nearly 80% could not correctly state that “hemorrhage” meant “bleeding”, “myocardial infarction” meant “heart attack”, or that “fractured” meant “broken”.11 This is despite the fact that greater than 50% of surveyed patients had a college education.

Studies have demonstrated that more patients comprehend PEMs written at lower grade reading levels. Overland et al.6 compared comprehension of diabetes education materials written at varied grade levels amongst 85 diabetic patients. Patients were randomized to read food care information at 6th, 9th, or 11th grade reading levels. 60% of patients assigned to 6th grade-level information understood it independently, whereas 21% understood the 9th grade-level information, and 19% understood the 11th grade-level information (p < 0.001). Baker et al.7 found that reducing the reading level of a patient education pamphlet on allergic contact rashes from an 11th to 7th grade level improved comprehension by patients in a private dermatology practice.

While there is evidence that PEMs written at 6th to 8th grade levels improve comprehension, few studies demonstrate the effect of readable PEMs on health-related behaviors. Jacobsen et al.5 showed that an educational pamphlet on the pneumococcal vaccine written at a 5th grade level and given to patients in an inner city primary care waiting room increased pneumococcal vaccination rates more than five-fold and patient-clinician communication about the issue approximately four-fold (P < 0.001), when compared to a control group who received a pamphlet on nutrition. It should be noted that the nutritional pamphlet was also written at a 5th grade level, and no comparison was made between pamphlets of different grade reading levels in this study.

While there is currently a paucity of data demonstrating the effects of improved readability on health outcomes, it follows logically that, in order for PEMs to be effective, patients must be able to comprehend them. Studies have shown that many PEMs available in primary care waiting rooms are written at levels significantly higher than the reading abilities of most patients.12,13 In 1990 Davis et al.12 found that only 6% of PEMs sampled from primary care waiting rooms were written below a 9th grade reading level. In 2004 Wallace et al.13 found that only 7% of the American Academy of Family Practice’s PEMS were written below an 8th grade reading level. Various subspecialties such as cardiology14,15, infectious disease,16 plastic surgery,17 dermatology,18 orthopedics,19,20 palliative care,21 oncology22,23 and obstetrics and gynecology24,25 have studied the readability of the PEMs utilized in their fields, and findings consistently indicate that 9th–12th grade reading levels are necessary to read and comprehend these documents.

Patient Education Materials at the Point of Care

President Obama’s 2009 stimulus package set the goal of a certified electronic health record (EHR) for each person in the U.S. by 2014.26 Currently, EHRs include links to PEMs, allowing clinicians to access and print them during a patient visit without navigating away from the patient’s record.27 Because institutions are tacitly endorsing the PEMs made available through the EHRs they purchase, it is crucial to assess their readability.

Another source of PEMs available to clinicians at the point of care is the National Library of Medicine (NLM). The NLM is free, widely accessible, and accredited by the U.S. government. To date, there are no studies assessing the readability of PEMs supplied through the NLM or EHRs. The goal of our study was to determine the readability of PEMs supplied though the NLM and a popular EHR. We hypothesized that the majority of these PEMs would be written at grade levels above the recommended target readability of a 5th–8th grade reading level.

DESIGN

Sample Selection

We included a convenience sample of EHR-linked PEM databases with significant market penetration and accessibility. Penetration was based on the number of hospitals using each EHR. Per a 2009 report by Modern Healthcare, Meditech is a popular EHR supplier, providing EHRs to 27% of US hospitals. McKesson Provider Technologies is the second most popular, providing EHRs to 14% of US hospitals, and Cerner Corp is the third most popular, providing EHRs to 13% of US hospitals.28 Meditech collaborates with three organizations to provide PEMs: Thomson and Reuters’ Micromedex, PatientEDU and EBSCO (also marketed as Lexicomp).29 Collaborations were based on the widespread use of these PEM databases by Meditech customers, as well as their capacity to integrate with Meditech’s technological platform. Because Micromedex and EBSCO are accessible in most academic hospitals, while PatientEDU requires a subscription, we sampled PEMs from the two former databases. We also sampled PEMs from MedlinePlus, the National Library of Medicine’s PEM supplier. This government-endorsed database has received numerous awards30 and is a commonly used web-based source of health information. It is accessible to anyone with internet access and has received over 155 million visits in 2010.31

To determine the percentage of PEMs written above target readability, we randomly selected 100 disease/condition-matched PEMs from each of three databases (n = 300 PEMs), and assessed their readability. The target of 100 PEMs was based on proportions assuming 20,000 total PEMs per database with 50% of PEMs above target readability, a 10% margin of error, and α = 0.05. PEMs were selected from a web-based alphabetical list via a random numbers generating program.32 If a disease topic was present in one PEM database but not in the others, the next alphabetical disease topic on the list was checked against the other two databases until a match was found. This methodology assured the avoidance of particularly obscure topics, and favored the inclusion of common diseases topics such as arthritis, cardiac angina, diabetes, gastroesophageal reflux, and low back pain. Titles, citations, glossaries, and “further resources” listed on each handout were excluded from the readability analysis. In MedlinePlus’ database, multiple resources are listed for each disease. “Patient handouts” and “Easy-To-Read” resources were selected whenever possible. If these were not available, handouts were taken from the “Medical Encyclopedia” or the first listing in the “Overview” category for a given disease. The “alternative names” category at the conclusion of these handouts was excluded from the readability analysis, as it often consisted of lists of medical terms without informational content.

MEASUREMENTS

Readability indices are used to assess health documents, including PEMs.33 They use mathematical formulas to assign passages of text a grade reading level based on word and sentence length (Table 1). Word length is a proxy for semantic or meaning difficulty, and sentence length is a measure of syntactic complexity.34

Table 1 Readability Indices
Full size table
Table 2 Readability Statistics
Full size table

Based on recommendations from the Health Literacy Advisor (HLA),34 we chose the Simple Measure of Gobbledygook (SMOG), Gunning Fog Index (GFI), and Flesch–Kincaid Index (FKI) for our analysis. SMOG is additionally recommended by the Centers for Medicare and Medicaid Services (CMS).35

Text from each PEM was copied and pasted into an online readability calculator36 recommended by the University of Minnesota Biomedical Libraries37 for aiding in the creation of patient education materials. This web-based tool analyzes the grade reading level of English text using a series of readability indices, including the three listed above. It also provides suggestions on how to improve readability by specifying sentences that increase the grade reading level of the text. Before the readability calculation, each PEM was cleaned to allow for accurate calculation. Cleaning involved standardizing the punctuation such that a period marked the end of each heading, sentence fragment, or sentence. The output from the online readability calculator was then copied into an excel spreadsheet for analysis. Data extraction was performed manually from MedlinePlus and Micromedex PEMs. Data extraction from EBSCO PEMs was performed by an automated computer program designed especially for this task. The program automated the method of manual data extraction. It ran through EBSCO’s PEM database, copied the text from each PEM, punctuated each line with a period, and pasted the text into the readability calculator. The program then recorded each PEM’s output in an Excel document. Automated data extraction was only possible for EBSCO due to the different formats of each database’s PEMs.

To determine the reliability of the computerized abstractions, two investigators (LS and NS) manually abstracted and cleaned ten randomly selected32 EBSCO PEMs, reanalyzed them using GFI, and determined the interrater reliability between the computer and manually abstracted PEMs through kappa analysis. To determine the inter-rater reliability of the manual data extractions for the Micromedex and Medline PEMs, a second investigator (NS) manually abstracted and cleaned 10 randomly selected PEMs and analyzed their readability, also using GFI. Additionally, we assessed reliability of our online calculator by entering the same ten MedlinePlus PEMs into a second online calculator38 to ensure inter-rater reliability between two online calculators using the GFI.

There was near perfect agreement between the computer program and manual extraction, manual extraction by authors, and two online readability calculators, with all kappas = 1. The mean errors were 0.17 grade levels between computer and manual extraction, 0.12 grade levels between manual raters, and 0.2 grade levels between online calculators.

Analysis

We determined the percentage of PEMs in each database written above the 8th grade level. Differences in the mean readability scores for PEMs across the three databases were determined with one-way analyses of variance (ANOVA) using SPSS version 19.0. We recorded sentences that raised the grade reading levels of the least readable PEMs in each database.

RESULTS

There was wide variability in the readability of PEMs from the three sources, but a majority were written above the 8th grade level (Table 2). The average reading level for MedlinePlus, EBSCO, and Micromedex PEMs was 10.2 (1.9), 9.7 (1.3), and 8.6 (0.9), respectively (p ≤ 0.000) as estimated by the GFI. Estimates of readability using SMOG and FKI were similar (Table 2).

The range of grade reading levels varied across databases. The most readable PEMs were consistently readable at a 5th–7th grade level in each database. EBSCO’s least readable PEMs were readable at 11th–13th grade levels, while MedlinePlus’ were readable at 14th–17th grade levels and Micromedex’s were readable at an 11th grade level. For each database, the percentage of PEMs above target readability, median grade reading levels, and ranges are shown in Table 2. In an analysis of average readability scores using the Gunning Fog index, Micomedex’s PEMs were more readable than MedlinePlus’ and EBSCO’s (p ≤ 0.000). MedlinePlus’ PEMs were more readable than EBSCO’ PEMs (p ≤ 0.039). PEMs assessed by FKI scored at an average grade reading level of 8.2—significantly lower than PEMs assessed by GFI (9.4) and SMOG (9.8) (p ≤ 0.000).

Eighteen out of 100 sampled disease or condition topics in MedlinePlus’ PEMs database were designated as “Easy-To-Read”. However, 72% of these 18 “Easy-To-Read” PEMs were a full grade level above target readability by the measure of at least one readability index.

PEMs on bacterial vaginosis, Barrett’s esophagus, generalized anxiety, malaria, pyelonephritis, salmonella, undescended testes, and yeast infections were at least two grade levels above target readability in all databases, as measured by the GFI. Examples of difficult-to-read sentences from each of these PEMs are listed in Table 3. Additionally, some disease topics were written at target readability in all databases. Examples of well-written, readable sentences from these three PEMs are listed in Table 4.

Table 3 Disease Topics Above Target Readability in All Databases
Full size table
Table 4 Disease Topics At Target Readability in All Databases
Full size table

DISCUSSION

Our study found that the majority of PEMs supplied through Meditech and the NLM were written above target readability. Micromedex’s PEMs are more readable than MedlinePlus’ or EBSCO’s by a significant margin. Our findings are consistent with results of studies in which investigators found only 6% to 7% of PEMs to be readable below an 8th grade level.12,13,39 While these studies assessed PEMs distributed in primary care waiting rooms12 and downloaded off the American Academy of Family Physicians (AAFP) website,13 our study is unique in that it examines the PEMs provided for clinician use at the point of care in EHR’s. Terry Davis, a health literacy expert, argued that having PEMs available during a physician visit would encourage physicians to use them as teaching tools, highlighting key points related to the topic of the medical encounter. Davis argues that this would be more meaningful for patients than simply having materials available in the waiting room.9

Studies have found a high level of correlation between various readability indices. For instance, Meade et al.33 found a 99% correlation between GFI and SMOG when assessing 49 health-based materials. In our analysis, a significant percentage of documents from each database were below target readability when analyzed using FKI, but above target readability when analyzed by SMOG and GFI. This is because FKI uses a lower criterion score than SMOG and GFI. While FKI predicts the level of reading skill required to correctly answer 75% of the questions on a reading test, GFI and SMOG predict the reading ability required to correctly answer 90-100% of the questions on a reading test.40 Both the Health Literacy Advisor34 and CMS35 warn that FKI underestimates grade reading level by approximately two grades. This likely explains the discrepancy.

As EHRs become the norm in practice settings, and efficiency is prioritized by busy clinicians, it is essential that the tools clinicians use to educate their patients be optimal and effective. Our results indicate that too many PEMs made available to physicians at the point-of-care are inadequate are written above recommended reading levels.

Limitations

Our study has several limitations. First, the 300 PEMs analyzed in this study are a random sample from three databases. We sampled PEMs from a national source, and an EHR with 27% market penetration. Our sample may not be representative of all of the PEMs, or the most commonly used PEMs. Second, the readability calculator used in this study only accommodates English text. Therefore, while each database provides non-English PEMs, these PEMs were excluded from our analysis. Third, results from our readability calculator may differ from results of other online readability calculators. Because calculators use different algorithms to count sentences, words, and syllables, there may be discrepancies between results, even when different calculators use the same formulas.40

Fourth, readability formulas are imperfect predictors of comprehensibility. They account for 50–84% of the variance in text difficulty as measured by comprehension tests.40 This is because readability indices do not account for some variables that influence comprehension and recall, such as visual aids, text organization, syntax, and rhetorical structure.41 Additionally, sentence and word length are not perfect measures of complexity. Multisyllabic words are not always unfamiliar, and short words may be difficult jargon. Our study does not assess how well patients understand the content of the PEMs examined.

Lastly, while it is necessary to improve readability of PEMs, this intervention alone may not be sufficient to improve health outcomes. Few published studies have shown a beneficial effect on patients’ health outcomes from using simplified reading materials alone.

CONCLUSIONS

The majority of PEMs available through Meditech and the NLM are written at reading levels higher than what has been recommended by expert groups. As EHR’s become ubiquitous and health care providers become increasingly pressed for time, the availability of accessible and comprehensible PEMs becomes more important. Improving patient comprehension of disease processes, prevention, and treatment is a necessary step towards improving health outcomes. Given health literacy’s enormous impact on our healthcare system, it is crucial that EHR vendors and the US government provide educational materials that are written at appropriate reading levels designed for comprehension by patients at all levels of health literacy. Further studies should assess the appropriateness of additional strategies for educating low health literacy patients, such as visual, animated, and interactive PEMs.