Abstract
Background
The National Institutes of Health (NIH) recommends patient education materials reflect the average reading grade level of the US population. Due to the importance of shared decision-making in breast cancer surgery, this study evaluates the reading level of patient education materials from National Cancer Institute-designated cancer centers (NCI-DCC) compared with top Internet search results.
Methods
Online materials from NCI-DCC and top Internet search results on breast cancer, staging, surgical options, and pre- and postoperative expectations were analyzed using three validated readability algorithms: Simplified Measure of Gobbledygook Readability Formula, Coleman–Liau index, and Flesch–Kincaid grade level. Mean readability was compared across source groups and information subcategories using an unpaired t-test with statistical significance set at p < 0.05. Mean readability was compared using a one-way analysis of variance.
Results
Mean readability scores from NCI-DCC and Internet groups ranged from a 9th–12th grade level, significantly above the NIH recommended reading level of 6th–7th grade. There was no significant difference between reading levels from the two sources. The discrepancy between actual and recommended reading level was most pronounced for “surgical options” at a 10th–12th grade level from both sources.
Conclusions
Patient education materials on breast cancer from both NCI-DCC and top Internet search results were written several reading grade levels higher than the NIH recommendation. Materials should be revised to enhance patient comprehension of breast cancer surgical treatment and guide patients in this important decision-making process to ultimately improve health outcomes.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Health literacy describes an individual’s ability to understand and use health information to empower healthcare ownership.1 Low health literacy has been associated with decreased preventative care (e.g., screening mammography, influenza vaccination), increased emergency care usage, increased hospitalizations, and increased post-surgical mortality.2,3 Patient populations previously identified with lower health literacy often have a lower education level, lower socioeconomic status, experience chronic disease, and/or are racial and ethnic minorities.4,5,6,7,8,9 Therefore, gaps between health information provided and patient health literacy has the potential to widen healthcare disparities.
Breast cancer remains the most common cancer diagnosis in women, with nearly 300,000 new cases of invasive breast cancer expected in 2023.10 There are several different surgical treatment options (lumpectomy versus mastectomy, with or without breast reconstruction) available for patients with resectable breast cancer. The variety of surgical options can complicate the decision-making process. To promote shared decision-making, the surgeon should provide treatment-related information that meets the health literacy needs of each patient.
Prior breast cancer literature exploring patient satisfaction with their surgeons found that physicians spend about 19 min before and after surgery with patients with breast cancer discussing diagnosis and next steps.11 Unfortunately, due to time constraints, patients often leave these appointments confused, with further unanswered questions that they may seek to clarify on the Internet. The 2009 National Health Interview Survey data found that 51% of women ages 18 years and over have used the Internet for health information.12 The Internet provides a platform for healthcare institutions, including National Cancer Institute (NCI)-designated cancer centers (NCI-DCC), to provide information regarding breast cancer surgery. In addition to academic resources available online, patients may access non-academic websites that publish information regarding breast cancer surgery. The use and applicability of such Internet-based information depends on the readability of that information. The National Institutes of Health (NIH) recommends that health materials for patients be written at a 6–7th grade reading level to meet the reading grade level of the average adult in the USA.13,14
While prior studies describe the importance of patient education materials on breast cancer, this study aims to determine if these materials are understandable and accessible to the average US adult. This study describes the readability of online patient education materials on breast cancer surgery in relation to the recommended reading level proposed by the NIH. In this study, we hypothesized that the readability of patient education materials on breast cancer surgery would be above the NIH recommended reading grade level for both NCI-DCC and top Internet search results.
Methods
Content Selection
NCI-DCC were identified using the NCI website (n = 71). Seven basic laboratory cancer centers and one pediatric center were excluded, leaving 63 NCI-DCC, termed the “NCI group.” To evaluate the accessibility of patient education materials on breast cancer to an Internet user, the number of clicks required to find the breast cancer information webpage from the homepage of each NCI-DCC was recorded.
Breast-cancer-related search terms were entered in the Google™ (Google, Inc, Mountain View, CA) search engine to identify and evaluate websites from top Internet search results, termed the “Internet group.” Search terms are listed in Table 1. Prior to each search, the browser was cleared, including location tools, user information, and cookies to minimize search bias. The search was performed in “incognito mode” using the Chrome browser (Google, Inc, Mountain View, CA) to avoid further search bias. The number of clicks needed to navigate to breast cancer information was not collected for the Internet group, as only information found within one click was analyzed. The first 60 search results of each search term were analyzed, excluding ads and any NCI-DCC websites.
For the NCI and the Internet groups, the following information subcategories were analyzed for readability: general breast cancer information, staging, surgical options, preoperative expectations, and postoperative expectations. The number of graphics, defined as tables, videos, images, was recorded for each group.
Readability Assessment
The text collected from each website was analyzed using three different validated readability algorithms: Simplified Measure of Gobbledygook Readability Formula (SMOG), Coleman–Liau index, and Flesch–Kincaid grade level (FKGL). The SMOG Readability Formula evaluates and analyzes the number of words with three or more syllables, along with the average number of sentences.15,16 The Coleman–Liau index analyzes the average number of letters and number of sentences per 100 words.17,18 The FKGL analyzes number of syllables per word and words per sentence.19,20 These readability scores are expressed as grade levels. Each respective readability score has been validated, with the SMOG Formula used most frequently in healthcare analyses.16,18,19,20,21,22,23,24,25
Statistical Analysis
Analysis was performed using Microsoft Excel (Microsoft Corporation, Redmond, WA) and JMP V.17 (SAS Institute, Inc, Campus Drive, Cary, NC). The mean readability grade level and standard deviation (SD) for each group (NCI and Internet) and information subcategory were determined with each readability algorithm. Mean readability was compared across groups and information subcategories using an unpaired t-test with statistical significance set at p < 0.05. Mean readability was compared using a one-way analysis of variance (ANOVA).
Results
NCI Group Resources
Of the 63 included in the NCI group, 57% of centers provided online general breast cancer information (n = 36), 48% provided staging information (n = 30), and 63% provided information on surgical options (n = 40). Fewer sites provided information on preoperative expectations (n = 11) or postoperative expectations (n = 17) (Table 2). Of the 21 centers with no information available online, 14 provided a link to external resources.
Readability
Overall, the readability of patient education materials from both the NCI group and the Internet group were well above the NIH recommended reading grade level of 6th–7th grade by each of the three readability scoring methods (Table 3). The mean reading grade level ranged from 9.0 to 11.2 in the NCI group and 9.4 to 10.9 in the Internet group, depending upon the information subcategory being discussed. Overall, no statistically significant difference was found in the means of all of the readability algorithms between the NCI and Internet groups. However, the mean readability by Coleman–Liau Index was found to be significantly higher in the NCI group (11.9) compared with the Internet group (11.1, p = 0.02) within the surgical options subcategory. In contrast, the mean readability by FKGL was significantly higher in the Internet group (11.2) than in the NCI group (9.2, p = 0.04) for breast cancer staging information.
Across all information subcategories examined, the mean readability of materials from the NCI group was significantly above the NIH recommended reading grade level. This difference between observed and recommended level was most pronounced for the surgical options subcategory, with a mean readability of 11.2 (11th grade reading level). Similarly, the mean readability of materials from the Internet group was above the NIH recommended reading grade level and was also most notable for surgical options.
Graphics and Accessibility
In all patient education materials assessed, the NCI group and Internet group had a mean of one graphic per page for each of breast cancer information, staging, preoperative and postoperative expectations. In contrast, for surgical options, the NCI group had a mean of two graphics per page and the Internet group had one. The mean number of clicks needed to navigate to the breast cancer page in the NCI group was three (SD 1.6). The number of clicks needed in the Internet group was not recorded and was one, given that the webpage was only included if it was the top result of the targeted search for each information subcategory.
Discussion
In this cross-sectional evaluation of readability of patient education materials on breast cancer, this study found that online materials are written at a reading level well above the NIH recommendation, regardless of the source. Subcategories including breast cancer information, staging, surgical options, preoperative and postoperative expectations all demonstrated mean readability between a 9th and 12th grade reading level for both the NCI group webpages and top Internet search results. The most pronounced disparity in readability between the information available online and the NIH recommendation was observed in materials discussing surgical options. Online preoperative and postoperative expectations education materials were not available from the majority of NCI-DCC websites. While previous studies have analyzed readability of education materials, this study is the first to characterize both readability and accessibility of patient education materials on breast cancer surgery.26,27,28,29
When provided information does not meet the health literacy needs of patients, shared decision-making becomes incredibly challenging. Patients with low health literacy are less likely to participate in treatment decision-making, are more likely to have decreased patient satisfaction, and experience higher rates of hospitalization, morbidity, and mortality.30,31,32 Lack of understanding and communication has also been shown to increase malpractice claims.33 The comprehension and alignment of provided healthcare information with the healthcare goals of the patient is critical in breast cancer surgery, where a plethora of treatment options are available. Accordingly, the surgical options subcategory demonstrated the highest readability scores from both the NCI and the Internet groups, approximately four grade levels above what is recommended. The higher readability scores may be due to the inherent complexity of the terms and techniques described in breast cancer surgery. Examples of this include terminology such as “nipple-sparing mastectomy” and “breast conserving therapy.” With the knowledge of how readability scales calculate grade level, using fewer syllables and shorter sentence lengths will improve the readability of the text.15,16,17,18 However, this may not always be possible when listing surgical procedures. Thus, providing patients with definitions for complex terms, such as “removal of the breast but keeping the nipple for appearance” for nipple-sparing mastectomy will help facilitate greater understanding. Without such, patients are more likely to turn to the Internet for information, which may lead to further confusion or miscommunication.34,35,36 Of note, a recent study has found that most individuals utilize the Internet right after cancer diagnosis versus during treatment and/or follow-up, exemplifying a need for reliable and accurate information to be given to the patient near the time of diagnosis.37
Delving further into the ability to navigate these websites, the number of clicks to get to the desired content was also analyzed in this study. Utilizing the Internet for healthcare information can be time consuming and may be difficult for patients to navigate.38 This study identified a mean of three clicks to navigate to the main breast cancer home page for NCI-DCC sites. From the home page, subcategories could further be accessed. For patients with access to technology, those with less technological experience may struggle to navigate to the main page.39,40 Providing patients with a direct website uniform resource locator (URL) or instructions on how to access reliable online patient education materials may reduce this technology barrier.
In addition to accessibility of online materials, the presence of graphics or other visual aids may provide greater comprehension of patient education materials.41,42,43 This study found that the average number of graphics for each subcategory of patient education materials was between one and two graphics. Although the readability algorithms utilized in this study do not include graphics within the calculation, prior studies have found that images, figures, and graphics are all useful tools to convey patient information and increase understanding of material content and ensure understanding in patients who may have reading difficulties.44,45,46 Furthermore, given the complexity of surgical terminology described previously, the inclusion of images may increase patient understanding of such procedures. A study by Hung and Stones found that patients with low health literacy had higher rates of satisfaction when materials included visuals.47 Specifically, typeface and color designs, as well as realistic photos, were associated with highest rates of satisfaction.47 These patients also preferred visual designs that took into account cultural factors. This aligns with other studies that have found that patients prefer graphics with similar demographics to the individual reviewing the materials.41 This emphasizes the importance of providing a wide range of images, illustrations, and videos that match the diversity of the patient population in breast surgery.48,49
Differences between the observed and recommended reading levels in online patient education materials have been similarly demonstrated in other surgical fields, including trauma, orthopedic surgery, vascular surgery, and plastic surgery.14,50,51,52 This current study further demonstrates the need for various surgical subspecialties, in this case breast surgical oncology, to revisit education materials to ensure patients understand their healthcare materials. This study demonstrates the need for revision and/or creation of online patient education materials on breast cancer that can be displayed on reliable and accurate websites such as NCI-DCC webpages.
This study has certain limitations as only free, online materials were assessed. It is possible that these centers have different written materials available for their patients that would not have been captured in this study. Materials may also be available online exclusively through patient portals and would not be accessible through a free online search without login credentials. Additionally, 14 of the 21 NCI-DCC websites without their own unique information included vetted links to outside education materials from other organizations. However, these were not included in our analyses due to the materials being created from organizations outside of the NCI-DCC center; providing patients with a list of trusted sources is another option for centers that may not be able to create patient education materials on their own due to limitations such as funding or staffing. Lastly, this study only evaluated materials written in English. Future studies should expand to evaluate patient education materials in different languages to best understand and characterize the readability of these materials.
Conclusions
Online patient education materials on breast cancer are written at several reading grade levels higher than the NIH recommended reading level, regardless of the source. To improve patient understanding of and participation in breast cancer treatment option decision-making, education materials should be revised for improved readability to empower and guide patients in this important decision-making process.
References
Rowlands G. Health literacy. Hum Vaccin Immunother. 2014;10(7):2130–5.
Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155(2):97–107.
De Oliveira GS, McCarthy RJ, Wolf MS, Holl J. The impact of health literacy in the care of surgical patients: a qualitative systematic review. BMC Surg. 2015;15(1):86.
Rikard RV, Thompson MS, McKinney J, Beauchamp A. Examining health literacy disparities in the United States: a third look at the National Assessment of Adult Literacy (NAAL). BMC Public Health. 2016;16(1):975.
Williams DR, Collins C. US Socioeconomic and racial differences in health: patterns and explanations. Ann Rev Sociol. 1995;21(1):349–86.
Jeppesen KM, Coyle JD, Miser WF. Screening questions to predict limited health literacy: a cross-sectional study of patients with diabetes mellitus. Ann Fam Med. 2009;7(1):24–31.
Gazmararian JA, Baker DW, Williams MV, Parker RM, Scott TL, Green C, et al. Health literacy among Medicare enrollees in a managed care organization. JAMA. 1999;281(6):545–51.
Ayotte BJ, Allaire JC, Bosworth H. The associations of patient demographic characteristics and health information recall: the mediating role of health literacy. Aging Neuropsychol Cogn. 2009;16(4):419–32.
Sudore RL, Yaffee K, Satterfield S, Harris TB, Mehta KM, Simonsick EM, et al. Limited literacy and mortality in the elderly: the health, aging, and body composition study. J Gen Intern Med. 2006;21(8):806–12.
Siegel RL, Miller KD, Wagle NS, Jemal A. Cancer statistics 2023 CA. Cancer J Clin. 2023;73(1):17–48.
Guy GP Jr, Richardson LC. Visit duration for outpatient physician office visits among patients with cancer. J Oncol Pract. 2012;8(3S):2–8.
Cohen RA. Use of the Internet for health information: United States. Natl Center Health Stat. 2011;66:1–8.
Hutchinson N, Baird GL, Garg M. Examining the reading level of internet medical information for common internal medicine diagnoses. Am J Med. 2016;129(6):637–9.
Powell LE, Andersen ES, Pozez AL. Assessing readability of patient education materials on breast reconstruction by major US academic hospitals as compared with nonacademic sites. Ann Plast Surg. 2021;86(6):610–4.
Mc Laughlin GH. SMOG Grading-a new readability formula. J Reading. 1969;12(8):639–46.
Wang LW, Miller MJ, Schmitt MR, Wen FK. Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm. 2013;9(5):503–16.
Betschart P, Abt D, Schmid HP, Viktorin P, Langenauer J, Zumstein V. Readability assessment of commonly used urological questionnaires. Investig Clin Urol. 2018;59(5):297–304.
Coleman M, Liau TL. A computer readability formula designed for machine scoring. J Appl Psychol. 1975;60:283–4.
Jindal P, MacDermid J. Assessing reading levels of health information: uses and limitations of flesch formula. Educ Health. 2017;30(1):84–8.
Swanson CE, Fox H. Validity of readability formulas. J Appl Psychol. 1953;37:114–8.
DuBay WH. The principles of readability. Costa Mesa: Impact Information, 2004.
Grabeel KL, Russomanno J, Oelschlegel S, Tester E, Heidel RE. Computerized versus hand-scored health literacy tools: a comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materials. J Med Libr Assoc. 2018;106(1):38–45.
Rudd RE, Anderson JE. The health literacy environment of hospitals and health centers. partners for action: making your healthcare facility literacy-friendly. Health and Adult Literacy and Learning Initiative. Harvard School of Public Health. 2006; https://www.hsph.harvard.edu/healthliteracy/practice/environmental-barriers/. Accessed 4/15/2023.
Brangan S. Development of SMOG-Cro readability formula for healthcare communication and patient education. Coll Antropol. 2015;39:11–20.
Kue J, Klemanski DL, Browning KK. Evaluating readability scores of treatment summaries and cancer survivorship care plans. JCO Oncol Pract. 2021;17(10):615–21.
Sand-Jecklin K. The impact of medical terminology on readability of patient education materials. J Community Health Nurs. 2007;24(2):119–29.
Agarwal N, Chaudhari A, Hansberry DR, Tomei KL, Prestigiacomo CJ. A comparative analysis of neurosurgical online education materials to assess patient comprehension. J Clin Neurosci. 2013;20(10):1357–61.
Ayyaswami V, Padmanabhan D, Patel M, Prabhu AV, Hansberry DR, Agarwal N, et al. A readability analysis of online cardiovascular disease-related health education materials. HLRP: Health Lit Res Pract. 2019;3(2):e75–80.
Eltorai AEM, Sharma P, Wang J, Daniels AH. Most american academy of orthopaedic surgeons’ online patient education material exceeds average patient reading level. Clin Orthop Relat Res. 2015;473(4):1181–6.
Kaye DR, Richardson CR, Ye Z, Herrel LA, Ellimoottil C, Miller DC. Association between patient satisfaction and short-term outcomes after major cancer surgery. Ann Surg Oncol. 2017;24(12):3486–93. https://doi.org/10.1245/s10434-017-6049-2
Chen Q, Beal EW, Okunrintemi V, Cerier E, Paredes A, Sun S, et al. The association between patient satisfaction and patient-reported health outcomes. J Patient Exp. 2019;6(3):201–9.
Rademakers J, Delnoji D, Nijman J, de Boer D. Educational inequalities in patient-centred care: patients’ preferences and experiences. BMC Health Serv Res. 2012;12:261.
Humphrey KE, Sundberg M, Milliren CE, Graham DA, Landrigan CP. Frequency and nature of communication and handoff failures in medical malpractice claims. J Patient Saf. 2022;18(2):130–7.
Wong C, Harrison C, Britt H, Henderson J. Patient use of the internet for health information. Aust Fam Phys. 2014;43(12):875–7.
Diaz JA, Griffith RA, Ng JJ, Reinert SE, Friedmann PD, Moulton AW. Patients’ use of the Internet for medical information. J Gen Intern Med. 2002;17(3):180–5.
Bujnowska-Fedak MM, Waligóra J, Mastalerz-Migas A. The internet as a source of health information and services. Adv Exp Med Biol. 2019;1211:1–16.
van Eenbergen MC, Vromans RD, Boll D, Kil PJM, Vos CM, Krahmer EJ, Mols F, van de Poll-Franse LV. Changes in internet use and wishes of cancer survivors: a comparison between 2005 and 2017. Cancer. 2020;126(2):408–15.
Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001;285(20):2612–21.
Early J, Hernandez A. Digital Disenfranchisement and COVID-19: Broadband internet access as a social determinant of health. Health Promot Pract. 2021;22(5):605–10.
Miller LM, Bell RA. Online health information seeking: the influence of age, information trustworthiness, and search challenges. J Aging Health. 2012;24(3):525–34.
Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient Educ Couns. 2006;61(2):173–90.
Sudore RL, Schillinger D. Interventions to improve care for patients with limited health literacy. J Clin Outcomes Manag. 2009;16(1):20–9.
Delp C, Jones J. Communicating information to patients: the use of cartoon illustrations to improve comprehension of instructions. Acad Emerg Med. 1996;3(3):264–70.
DeWalt DA, Broucksou KA, Hawk V, Brach C, Hink A, Rudd R, et al. Developing and testing the health literacy universal precautions toolkit. Nurs Outlook. 2011;59(2):85–94.
Norris EM. The constructive use of images in medical teaching: a literature review. JRSM Short Rep. 2012;3(5):33.
Dowse R, Ramela T, Barford KL, Browne S. Developing visual images for communicating information aboutantiretroviral side effects to a low-literate population. Afr J AIDS Res. 2010;9(3):213–24.
Hung YL, Stones C. Visual design in healthcare for low-literate users - A case study of healthcare leaflets for new immigrants in Taiwan. Interacción. 2014.
Smith RM, Anderson ES, Powell LE, Schuth OA, Mountziaris PM, Feldman MJ. It’s not all white: implicit racial bias in imagery used in plastic surgery resident education. J Surg Educ. 2022;79(4):943–9.
Hirko KA, Rocque G, Reasor E, Taye A, Daly A, Cutress RI, et al. The impact of race and ethnicity in breast cancer-disparities and implications for precision oncology. BMC Med. 2022;20(1):72.
Eltorai AE, Ghanian S, Adams CA Jr, Born CT, Daniels AH. Readability of patient education materials on the American Orthopaedic Society for Sports Medicine website. Phys Sportsmed. 2014;42(4):125–30.
Stocco F, Kwan JY, Sood M, Scott DJA, Bailey MA, Coughlin PA. Assessment of available online website and youtube resources for patients with abdominal aortic aneurysms. Ann Vasc Surg. 2023;S0890–5096(23):00252–62.
Eltorai AE, Ghanian S, Adams CA Jr, Born CT, Daniels AH. Readability of patient education materials on the american association for surgery of trauma website. Arch Trauma Res. 2014;3(2):e18161.
Acknowledgments
The authors have no acknowledgements.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Presentation: This study was presented at the Society of Surgical Oncology 2023 International Conference on Surgical Cancer Care in Boston, MA (3/23/23).
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rauzi, A., Powell, L.E., White, M. et al. Readability Analysis of Online Breast Cancer Surgery Patient Education Materials from National Cancer Institute-Designated Cancer Centers Compared with Top Internet Search Results. Ann Surg Oncol 30, 8061–8066 (2023). https://doi.org/10.1245/s10434-023-14279-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1245/s10434-023-14279-5