Abstract
Medical knowledge is increasing at an exponential rate. At the same time, unexplained variations in practice and patient outcomes and unacceptable rates of medical errors and inefficiencies in health care delivery have emerged. Our Institute for Health Care Delivery Science (I-HDS) began in 2014 as a novel platform to conduct multidisciplinary healthcare delivery research. We followed ten strategies to develop a successful institute with excellence in methodology and strong understanding of the value of team science. Our work was organized around five hubs: 1) Quality/Process Improvement and Systematic Review, 2) Comparative Effectiveness Research, Pragmatic Clinical Trials, and Predictive Analytics, 3) Health Economics and Decision Modeling, 4) Qualitative, Survey, and Mixed Methods, and 5) Training and Mentoring. In the first 5 years of the I-HDS, we have identified opportunities for change in clinical practice through research using our health system’s electronic health record (EHR) data, and designed programs to educate clinicians in the value of research to improve patient care and recognize efficiencies in processes. Testing the value of several model interventions has guided prioritization of evidence-based quality improvements. Some of the changes in practice have already been embedded in the EHR workflow successfully. Development and sustainability of the I-HDS has been fostered by a mix of internal and external funding, including philanthropic foundations. Challenges remain due to the highly competitive funding environment and changes needed to adapt the EHR to healthcare delivery research. Further stakeholder engagement and culture change working with hospital leadership and I-HDS core and affiliate members continues.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Medical knowledge is increasing at an exponential rate, reshaping the health care environment with a steady stream of diagnostic and therapeutic innovations. These advances offer unprecedented opportunities to improve health and reduce the disease burden of patients. At the same time, they often disrupt traditional care processes and put stresses on the health care delivery system. Numerous studies document a worrisome rate of medical error, inefficiency, and variations in practice patterns [1,2,3]. Addressing these problems requires a health care delivery system characterized by continuous learning and improvement [4, 5]. The path towards developing such a learning health system involves an iterative process of analysis of data from patients and care processes within the system, benchmarking to national standards, and designing and evaluating interventions with engagement from the appropriate stakeholders. This approach, known as health care delivery research (HCDR), leverages the expertise of scientists, researchers, and medical care providers with data from electronic health records (EHRs) for research to inform—and optimize—frontline clinical care. Optimization of clinical care delivery results in improved outcomes and cost savings. It also promotes the translation of research findings into practice through EHR implementation and stakeholder engagement while promoting careers of engaged providers through collaborative publications and research grants. In the U.S. as well as abroad, institutes focused on health care delivery research is typically part of academic institutions (e.g. Trinity College Dublin’s Centre for Health Policy and Management, Indian Institute of Health Management Research, and University of Canberra Health Innovation Precinct) or hospital based health organizations (e.g. Monash Health’s Centre for Clinical Effectiveness, German Center for Health Research, and Norwegian Center for Health Services Research).
The I-HDS at the Icahn School of Medicine at Mount Sinai was launched in 2014. At the same time, the Mount Sinai Health System was established, which merged seven hospitals with our medical school. The Institute’s charge was performing HCDR, and it received seed funding to support interdisciplinary full-time faculty and staff.
The I-HDS team began by surveying websites of all 134 accredited schools of medicine in the United States to find established institutes and programs that performed HCDR, to identify the essential foundational strategies for developing a successful institute (see Electronic Supplementary Material Table 1). Although 38 institutions showed some entity that performed HCDR, 15 of these websites were most relevant to our goals (see Electronic Supplementary Material Table 2). We reviewed organizational charts; the extent and types of collaborations among faculty, staff, and trainees; approaches to stakeholder engagement; the balance between institute team-led projects and those led by collaborators; financial models for providing and administrating services (e.g. consultations, collaborations); the types of projects undertaken; and the types of trainees and training activities. Following our guiding principle of developing a unit with expertise in various quantitative and qualitative methodologies, being open to collaboration with all clinical fields, and being dedicated to engage stakeholders at all level of staff and faculty, we identified ten foundational strategies and five thematic “hubs” for developing an institute to conduct HCDR.
Here, we briefly outline these foundational strategies (Fig. 1), share our experience around development and implementation, and describe the “hubs” (Fig. 2), created to support the institute’s infrastructure and development of our personnel. We report on our successes, share the lessons learned, and describe challenges. The essential strategies described here, and our hub-based approach, can be adapted, as appropriate, for health care systems of various sizes and composition.
1 Foundational strategies for developing a HCDR institute
1.1 Alignment of institute projects with institutional priorities —participation in health system and hospital committees
Ideally, a HCDR institute is dedicated to projects focused on improving delivery of patient care, rather than on traditional biomedical research. Such an institute will take its lead from hospital leadership in setting the research agenda in alignment with hospital priorities. Our I-HDS director or her designee meets regularly with hospital leadership and also sits on relevant hospital committees (e.g. Quality Leadership, Healthcare Acquired Infection, Falls, Readmission, Mount-Sinai Health System ‘Big Data’ Steering Commitee, Clinical Data Science Group). Once an issue is identified with potential for benefitting from research, we work closely with designated clinical and biomedical informatics champions to further define the data extraction and coding needed to implement the project within the EHR system. For example, when a hospital committee identified an increase in imaging orders, a clinical decision support (CDS) initiative was launched to assess the appropriateness of these orders and was subsequently implemented in our EHR. Institute personnel regularly monitored and analyzed use of the CDS, and then worked with informatics colleagues to fully incorporate it into workflows. We improved the CDS scoring system through informal usability testing with clinical providers and initiated a cluster-randomized clinical trial to assess the value of CDS in increasing appropriateness scores, as well as an observational study for examining its value in educating physicians-in training [6, 7].
1.2 Recruitment and professional development of institute core members
A successful HCDR institute should recruit its full-time core members from different backgrounds, including clinical epidemiology, biostatistics, decision science, health economics, health services research, medical sociology, and informatics research, among others. The core members’ diverse methodologic skills (e.g. statistical modeling, cost-effectiveness analysis, qualitative and survey research) are needed to address most applied research problems. It is useful to recruit individuals with broad interests across the major areas of HCDR to optimize their efficiency, output, and effectiveness. To foster recruitment and retention, our institute appoints faculty through our medical school’s Department of Population Health Science and Policy. We offer various tracks for faculty and graded levels of promotion for staff. Tenure-track faculty can choose independent investigator, collaborative research investigator, or clinical educator designations. Non-tenure track faculty can opt for collaborative research investigator or clinical educator designations. Alternatively, the institute can engage other faculty in different departments by paying portions of their salaries.
While HCDR research is different from traditional biomedical research, policies around promotion for faculty are likely the same. To support career development, our institute provides core members with protected time for project development and proposal writing, learning about emerging methodologies and new datasets, attending professional conferences, and participating in leadership development programs, among other career development activities. We evaluate institute personnel annually and make adjustments to assignments and roles, as necessary, to ensure that all members can be productive as they work in a setting (i.e. hospital environment) not traditionally aligned with advancing a research career.
1.3 Hospital stakeholder engagement through affiliate membership: Easing implementation and increasing impact
Our I-HDS director sent a letter to the members of every hospital committee inviting them to become “affiliate members,” detailing the opportunities and benefits of collaborating with the institute and asking them to identify others who might wish to collaborate [8]. Defining a cadre of affiliate members has helped us to find and engage appropriate stakeholders when projects are identified and develop collaborative initiatives. We host a bimonthly seminar series where affiliate members introduce their work and research problems they would like to address. Many of these presentations have led to fruitful collaborations, with several papers now under review. The image use/CDS project described above, for example, evolved out of a collaboration with affiliate members in the Departments of Radiology, Emergency Medicine, Information Technology (IT) and the Data Warehouse. This project was easier to implement in the EHR due to the involvement of IT personnel, and we could monitor data continuously due to our collaboration with personnel at the Mount Sinai Data Warehouse. This work made a significant impact in the institution by increasing the percentage of imaging orders that were designated as ‘appropriate’. Required workflow was changed because involved radiologists and Emergency Medicine doctors added this module to their practice guidelines and training. This experience illustrates the importance of identifying and engaging key stakeholders (who will differ based on the institution and the topic) early in creating an HCDR-focused institute.
1.4 Supporting programmatic collaborations
Successful HCDR institutes should establish programmatic collaborations with clinical institutes, Centers of Excellence, and divisions/departments. Institute personnel should have an allocated amount of effort devoted to such collaborations to sustain relationships and ensure productivity [9]. In addition to our collaborations with multiple departments (e.g. Orthopedics, Medicine, Neurology, Geriatrics) and institutes (e.g. cancer, respiratory disease, and addiction), we work directly with hospital support units (e.g. supply chain, pharmacy, clinical data science group).
1.5 Building a sustainable financial structure
Careful planning for institute sustainability is needed from the start and should be revisited annually. Our institute initially received seed funding for a team of five members (~$3 million for 4 years). We developed an effective long-term plan of sustainability through collaborative contracts. The departments, units, centers, and institutes with whom we engage contribute partial salary support for institute personnel through grants and contracts. We have also helped I-HDS core members develop proposals as principal investigators and created joint appointments with other departments. Through these approaches, I-HDS has expanded over the last five years to 20 members, as its support has transitioned to a higher proportion of external funding.
1.6 Aligning science and informatics
Access to large volumes of reliable data is critical in HCDR. It is important to know how and where patient data are stored (i.e. EHR system, data warehouse), how they can be accessed, and who must be engaged in data retrieval and extraction. It is helpful to build strong collaborative relationships with EHR and data warehouse personnel, groups/units engaged in high-quality data reporting at a national level, and departments collecting and storing genetic/genomics data to guide research in precision medicine. In our institute, we have jointly recruited members with data informatics groups who assist in improving the quality of data extraction and ensuring reproducibility. Our collaborations help speed the translation of findings into publications, new projects, and—more importantly—clinical practice [10, 11]. We frequently disseminate our successes through the EHR system and by training a multi-disciplinary care team so improvements are quickly implemented, scaled, and sustained long-term [12,13,14].
1.7 Training institute core members in health care delivery research
Institute core members will likely come from diverse disciplines and have varied exposure to health care delivery research. Thus, it is important to develop a strong training program that includes learning about hospital operations and challenges, as well as HCDR research methods. I-HDS personnel attend appropriate hospital meetings; conduct literature searches to understand the evidence base that supports specific practices and solutions and share these results in weekly institute meetings; participate in training related to the EHR; and take online courses. Our seminar series invites experts from our institution and others to inform our members about how different methods can be applied to emerging HCDR questions.
1.8 Educational activities and mentoring of clinicians and health care providers
Institute members teach courses in the graduate school on research methods and one-day immersion courses. We also mentor medical students, residents, fellows, and early-stage faculty in research and academic writing. Enabling frontline clinicians to see how data are used to answer important questions about care can result in collaborative publications, which increases clinicians’ commitment to conducting HCDR. Toward this end, I-HDS members present grand rounds in the hospital and clinical departments, share information about methods and study design, and offer mentoring and training in manuscript writing and data analysis. We also provide training on research related to specific patient care issues in our hospitals (e.g. evaluation of the effectiveness of infection control interventions, or overuse of laboratory testing) [14, 15].
1.9 Maintaining an effective and informative website
A well-designed, informative website is important to enhance external and internal visibility and can help an institute recruit new members and potential collaborators. A comprehensive site should provide information on personnel, current and past projects, programmatic collaborations, financial models, policies, and information about training and educational programs. All institute communications should include a link to the institute’s website [16].
1.10 Project tracking with mandatory data input fields
Developing an e-tracking project system with mandatory input fields is an effective method of accurately logging and tracking collaborative projects. The REDCap-based system we developed in-house tracks work patterns for reporting our productivity both to collaborators and our institution. This system has helped us to generate the data to create, and adjust, requests for collaborations in a formatted template. It also allows us to track core members’ successful projects and publications, which we regularly disseminate. We recommend, and will soon implement in I-HDS, an “Investigator Satisfaction Survey” to support continuous quality improvement.
2 Organizing hubs
Health care delivery science is complex, requiring faculty from a variety of backgrounds who are supported by strategic partners across the institution. To facilitate and streamline this complex effort, we have organized our infrastructure, personnel, and research around five “hubs”in Fig. 2. Institute projects are classified in at least one hub, which provides a theoretical and practical framework for activities. Hubs should be based on an institute’s priorities; however, they may emerge after the initial years of development. Here we describe the 5 hubs most salient to I-HDS.
2.1 Quality/process improvement and systematic review
This hub’s goal is to provide a knowledge base for systematic review of issues confronting the hospital management team and initiation of projects to improve quality of care or process of care delivery through informatics interventions (e.g., CDS, educational interventions, or both) [17]. Included in this hub, for example, is our study on high-risk medications in hospitalized elderly adults in one of our hospitals. In that study, on reviewing 328 falls as recorded in the EHR, we found that 60% of older patients who fell had received high-risk medication in the preceding 24 h – with some administered at higher-than-recommended daily doses. Our findings suggested that decreasing default doses for individuals aged 65 could decrease inpatient falls. As a result, our health system adopted a policy of rigorous medication review and set the default dose to a lower level in the EHR system [17]. Adoption of this change needed engagement with all types of providers taking care of elderly patients, and approval by the health system’s Falls Committee. This change has become permanent in the system and number of falls have remained lower than before. We also executed projects evaluating interventions for reducing unnecessary use of drugs, laboratory and pathology tests, decreasing incidence of catheter-associated urinary tract infection, and getting patients discharged before noon [13,14,15, 18, 19]. Based on this evidence, these interventions have become part of the routine care and have created value for our patients and health system.
2.2 Comparative effectiveness research, pragmatic clinical trials, and predictive analytics
Comparative effectiveness research, randomized clinical trials (RCTs), pragmatic clinical trials (PCTs), RCTs with stepped-wedge and cluster designs, and predictive analytics are all valuable approaches in HCDR. It is important to use each based on the question to be answered and its feasibility in the particular setting [20, 21]. For example, I-HDS collaborated on a study of cancer registry data demonstrating the effectiveness of adjuvant chemotherapy versus observation post-cystectomy in patients with locally advanced bladder cancer. The results have impacted patient counseling around treatment decisions and led to creation of a web-based shared decision-making tool [22].
We also initiated a clustered randomized trial for assessing the value of CDS for high-cost imaging. Unfortunately, the study was terminated due to contamination of randomization from mislabeling of physicians in the EHR system. However, we collaborated with Aurora Health in Wisconsin to successfully execute the study in their system. This study impacted the decision of the Center for Medicare and Medicaid Services (CMS) to require use of CDS in determining appropriateness of high-cost imaging at the time of order placement [6, 7]. All health systems, including ours, have adopted use of CDS due to this requirement.
The I-HDS also has collaborated with our hospitals’ Clinical Data Science Group on development and manuscript writing of EHR data pipelines for machine learning-based analytic models to predict falls, clinical deterioration on the hospital floor, malnutrition, delirium, oncology care model, and sepsis [23,24,25,26,27,28]. Most of these models are currently being tested for their value in pragmatic trials. However, in one such project – having registered dieticians see high-risk patients before other patients – has already resulted in an increased percentage of malnutrition diagnoses, improving patient care. In another project, predicting which inpatients are likely to deteriorate in the next 6 h and sending alerts to a rapid response team (who take needed actions to avoid sending such patients to the ICU) has shown early signs of reduced number of ICU days and thereby increased overall availability of ICU beds [26].
2.3 Health economics and decision modeling
Health economics (HE) is concerned with issues related to efficiency, effectiveness, value and behavior in the production and consumption of health and health care and decision modeling (DM) adds a structure by which a variety of approaches could be compared under different parameters. Operations research (OR) is a related field of study that includes methods that can help address complex real-world healthcare problems by balancing the tradeoff between outcomes and available resources to maximize the overall benefit. All of these approaches may influence clinical decision-making – and more broadly, also may influence allocation of health care resources [27,28,29,30]. Examples of studies in this hub include cost-effectiveness analysis of using medication to prevent cardiovascular disease (CVD) as measured by CVD event, costs, and quality-adjusted life years (QALYs); health and economic impact of practices for total knee replacement, routine pathologic examination of removed tissue for primary shoulder arthroplasty, and intravenous acetaminophen use in pain control; and how diabetes status influences associations among postoperative hyperglycemia and clinical and economic outcomes in cardiac surgery [18, 19, 31,32,33,34,35,36] Details on the methodological approaches are available in related publications (see Electronic Supplementary Material Table 3).
Papers using health economics, decision modeling, and operation research techniques are generally difficult to publish, since clinical journals do not always have appropriate reviewers available. Some editors and reviewers may feel that results from modeling studies are not transparent and thus insufficiently reliable. However, because our desired readers are clinicians and policy makers, health economics journals are often not the best choice, since these typically focus on innovation of methods. To get these papers accepted for publication in journals targeted at a clinical audience, our approach has been to be persistent, write these manuscripts in a straightforward style, suggest reviewers with appropriate expertise, and include methodologic details as supplemental material. This approach has resulted in our work being published in well-known clinical journals (see Supplemental Table 3).
Results of these studies have also changed practice in several ways. For example, our orthopedics department now gathers patient-reported outcome data longitudinally for patients with knee osteoarthritis, so decisions regarding the need for total knee replacement can be based on the model described in our publication [31]. Similarly, clinicians in our health system weighing whether to use intravenous acetaminophen pre- and post-surgery now consider how this practice might interact with opioids, and optimal dosing patterns for dosing and timing, as found in our research [37].
2.4 Qualitative, survey, and mixed methods research
Qualitative research contributes to HCDR by providing insights into complex healthcare delivery issues not easily captured by quantitative data [38]. Qualitative approaches incorporate information obtained via direct observation (ethnography), interviews, focus groups, social media platforms, and community-based participatory research techniques [39,40,41]. I-HDS’s Qualitative, Survey, and Mixed Methods hub also includes development, use, and validation of new survey instruments, as well as adaptations of existing instruments for new purposes [42, 43].
In one qualitative research project, we observed critical care physicians’ and other providers’ communication while recording interruptions, patient safety events, and EHR use over 6 weeks. The aim was to describe the types, frequencies, and impact of ICU interruptions on patient safety event occurrences and EHR use, with the goal of reducing interruptions to improve care [44]. In another project, I-HDS’s medical sociologist and a multi-disciplinary team of clinicians participated in structured interviews regarding development of a medical home for patients with inflammatory bowel disease. Results of the interviews enhanced team building and are informing approaches to building this novel medical home [45].
2.5 Training/mentoring
The purpose of the Training/Mentoring hub is to evolve educational opportunities and educate stakeholders across our institution about the importance and relevance of HCDR. We are creating a Health Care Delivery Fellowship, to give frontline providers protected time to work on HCDR projects of their choosing. Members of this hub developed formal courses in our graduate school’s Master of Science in Health Care Delivery and Leadership program; and created and teach qualitative research, biostatistics, meta-analysis, decision science, and informatics tool evaluation courses in graduate programs in Clinical Research. The I-HDS also trains and mentors medical students, graduate students, residents, fellows, early-stage faculty, and other providers (e.g. pharmacists, infection disease practitioners, and informatics technicians) in study design, analysis, and reporting. These individuals have collaborated with their mentors on numerous publications [11, 15, 17, 37, 46, 47]. Our goal is to foster the engagement of these mentees in HCDR work as they obtain competitive internships, awards, admission to medical school, and career advancement.
3 Discussion – successes, challenges, and solutions in the first five years
Since the institute’s creation, the number of core members has increased from 5 to 20, and we have recruited about 100 affiliate members. Importantly, we created an organizational and financial structure that supports collaboration among all members. These teams have written collaborative grant applications resulting in awards totaling ~$15 million. The value of projects implemented in our health system is estimated at ~$20 million ($12 million for the malnutrition project and $8 million for the clinical deterioration project). Furthermore, I-HDS members contributed to over 100 publications describing our research and resulting changes in health care delivery in our hospitals and beyond [4, 6, 7, 10,11,12,13,14,15, 46,47,48].
Selected media mentions and feedback from external collaborators and patients have demonstrated the value of these projects. I-HDS members (CC and MM) led a project with members from seven Clinical Translational and Science Award (CTSA) institutions and developed a questionnaire for supporting data reproducibility in a learning health system. This project obtained sponsorship from CTSA Informatics executive committee (EC), Clinical Data to Health (CD2H), and Healthcare Data Analytics Association and surveyed appropriate representatives about clinical data quality-related practices, assessed awareness and perspectives on related issues, and gauged training needs [43]. Dr. Adam Wilcox, CD2H Co-Program Director, commented that “As this information becomes available, it will be important in informing the national data quality metrics development efforts.” Another project provided a year-long Spanish language-preferred patient navigation services for MyChart and OpenNotes (ON) “NotasAbiertas Para Todos: OpenNotes For All”, which also engaged family caregivers. The program received great reviews from participating patients, eliciting many positive comments: “Magnificent! I appreciate the program in Spanish and your patience for explaining everything so well and clearly”; “During this difficult time for immigrants, this is such a wonderful way to show that Mount Sinai really does care about us!” This project received grant support from New York State Health foundation and OpenNotes founders at Beth Israel Deaconess Medical Center - Harvard have called this program “ground-breaking”. The GRITT-IBD™ program (Gaining Resilience Through Transitions for patients with Inflammatory Bowel Diseases) members including I-HDS member (KG) won a team science award for working collaboratively across various disciplines from gastroenterology/medicine, nursing, clinical pharmacy, nutrition, social work, child life, population health, and behavioral/psychological health, to provide the best patient care using a team-based, patient-centered approach (sponsored by Mount Sinai Clinical and Translational Award Program).
Return on investment for I-HDS derives from research to support cost containment in imaging, laboratory, and pathology testing, and drug use; better management of personnel time to increase diagnoses of comorbidities that improve care and control costs; improving safety and quality in infection control and timely discharge with desired destination; nurturing academic collaborations with clinical faculty that have resulted in new projects, publications, and promotions; demonstrating to payers that our health system’s practices are evidence-based and that we achieve high-quality metrics; facilitating strategic leveraging of the hospital setting and data from the EHR in grant applications; and promoting the reputation of the health system and our medical school nationally and internationally. Aligning science and informatics has been a key ingredient to our success and has helped with joint recruitment of personnel. Our organization and financial structure, with specific percentages of I-HDS personnel being supported by the collaborating department, has enabled us to grow and to create a plan for long-term sustainability.
We have, nonetheless, encountered barriers during the initial years of the institute. For example, while we have many clinical champions and projects supported by their departments, their heavy clinical workload often makes it difficult for them to complete collaborative research projects. The I-HDS responded by providing protected time for senior faculty and staff members through its seed funding to work on such team-based projects and ensure their timely completion by buying back time from the clinical researchers’ other collaborative contracts [49].
Funding opportunities specific to HCDR are still limited and the competition for such support is intense. Thus, the I-HDS has broadened its grant-writing targets by creating relationships with philanthropic foundations through our institutional development office. We have also supported existing NIH-supported centers and played a key role in bringing an NCI designation to our cancer center.
We have not always had access to charge and cost data for our projects, which makes it difficult to estimate their full return on investment. We have now added personnel with financial and business expertise to our staff, and we continue to work with hospital leadership to make cost and charge data more accessible for our research and for showcasing our value.
Working across disciplines can reveal different styles and preferences in communication and collaboration. The I-HDS seminar series and ‘Getting to know what you do’ sessions presented by clinicians were helpful in improving these areas. Once embedded research in clinical practice began, improved understanding and buy-in came about. These collaborations ultimately led to quality and process improvement projects and fostered career development. Mentoring and training opportunities for all healthcare providers, including medical and graduate students, in these collaborative projects created a stimulating environment for learning and provided measurable success.
As others have found, using EHR data for research brings multiple challenges. Much data in such systems are captured in an unstructured format, and availability of personnel for data extraction and warehousing are often limited. Technology underlying data warehouse structures also becomes outdated very quickly, which affects the quality of the extracted data. In addition, designs frequently used in research, such as randomization, are difficult to incorporate into EHR systems. In response to these issues, our medical school created a new position of Chief Research Informatics Officer (CRIO). The CRIO is helping our I-HDS advance efforts to optimize research-related uses of clinical data, for projects with an HCDR or precision medicine emphasis.
The goal of writing this kind of paper is to share the experience from early years so others could emulate the good practices. We arrived at our guiding principle, 10 strategies, and 5 hubs through discussion with our institutional leaders, loosely put together as an ‘Internal Advisory Board (IAB)’, and fine-tuned by regular input from a formally put together ‘External Advisory Board (EAB)’. IAB consisted of dean, Chief Medical Officer (CMO), hospital presidents, clinical department chairs and vice chairs whereas EAB consisted of experts from the fields of population health, health services research, insurance company, and informatics research. Since many clinical institutes with focus on particular disease systems (e.g. Respiratory Institute, Critical Care Institute, Diabetes Institute etc.) already existed at our institution, it was recommended that I-HDS not focus on particular disease systems but rather focus on developing methodology strengths that will benefit all clinical institutes and the entire health system. As collaborations developed, depending on the methodology needs of these projects as well as the training and mentoring requests of personnel involved, we progressively created the five hubs: 1) Quality/Process Improvement and Systematic Review, 2) Comparative Effectiveness Research, Pragmatic Clinical Trials, and Predictive Analytics, 3) Health Economics and Decision Modeling, 4) Qualitative, Survey, and Mixed Methods, and 5) Training and Mentoring. For example, the project where the institute ultimately initiated a clustered randomized trial for assessing the value of clinical decision support (CDS) for high-cost imaging (as part of Hub 2) began as a quality improvement project needing systematic review (Hub 1) and it needed usability testing and focus group for development of the CDS system (Hub 4). The project was formulated by collaborating with key hospital stakeholders (S2) in an effort to align with institutional priorities of moving towards ‘appropriate’ use (S1) and trained fellows in clinical research (Hub 5). The process was organic, not pre-specified, and somewhat depended on the expertise of the personnel we were able to recruit.
The strategies and approaches outlined here can be adapted and adjusted for health systems and health centers of various sizes and with distinct clinical programs and institutional priorities. Diverse units devoted to some form of HCDR exist nationwide. For example, New York University School of Medicine developed an academic department with four expansive key missions of engaging community, turning information into insight, transforming healthcare, and shaping policy. The Johns Hopkins University School of Medicine opened an academic institute focused on patient safety and quality improvement through research, training, and practice. The University of Pittsburgh Medical College opened a comparative effectiveness research center to respond to the rising research interest and funding in patient-centered outcomes research. Emory University Medical School created a center for health discovery and well-being with emphasis on preventive health services and related research and training [50,51,52,53]. Each of these required quite different organizational approaches and assessments of how ‘value’ is measured. Institutions interested in developing new HCDR units would benefit from crystallizing their vision and mission according to their institutional needs and incorporating select features from other established institutions to inform planning and execution. A similar approach has helped inform successful development of the Clinical and Translational Science Institute’s Core Research Facilities and the National Cancer Institute-designated Cancer Biostatistics Shared Resources [54, 55].
References
Anderson JG, Abrahamson K (2017) Your health care may kill you: medical errors. Stud Health Technol Inform 234:13–17. https://doi.org/10.3233/978-1-61499-742-9-13
Kyle M, Williams H (2017) Is American health care uniquely inefficient? Evidence from prescription drugs. Am Econ Rev 107(5):486–490. https://doi.org/10.1257/aer.p20171086
Riley WJ (2012) Health disparities: gaps in access, quality and affordability of medical care. Trans Am Clin Climatol Assoc 123:167–172
Abraham E, Blanco c, Lee CC, Christian JB, Kass N, Larson EB, Mazumdar M et al. (2016) Generating knowledge from best care: advancing the continuously learning health system. NAM Perspectives. Discussion Paper, National Academy of Medicine, (Washington, DC):1-23. https://doi.org/10.31478/201609b
Guise JM, Savitz LA, Friedman CP (2018) Mind the gap: putting evidence into practice in the era of learning health systems. J Gen Intern Med 33(12):2237–2239. https://doi.org/10.1007/s11606-018-4633-1
Doyle J, Abraham S, Feeney L, Reimer S, Finkelstein A (2019) Clinical decision support for high-cost imaging: a randomized clinical trial. PLoS One 14(3):1–13. https://doi.org/10.1371/journal.pone.0213373
Poeran J, Mao LJ, Zubizarreta N, Mazumdar M, Darrow B, Genes N, Kannry J, Francaviglia P, Kennelly PD, Whitehorn J, Kilroy G, Garcia D, Mendelson DS (2019) Effect of clinical decision support on appropriateness of advanced imaging use among physicians-in-training. AJR Am J Roentgenol 212(4):859–866. https://doi.org/10.2214/AJR.18.19931
Mallery C, Ganachari D, Fernandez J, Smeeding L, Robinson S, Moon M, Lavallee D, Siegel J. (2012) Innovative methods in stakeholder engagement: an environmental scan. http://www.effectivehealthcare.ahrq.gov/index.cfm/tools-and-resources/how-to-get-involved-in-the-effective-health-care-program/. Accessed 4/30/2020
Perkins SM, Bacchetti P, Davey CS, Lindsell CJ, Mazumdar M, Oster RA, Peduzzi PN, Rocke DM, Rudser KD, Kim M (2016) Best practices for biostatistical consultation and collaboration in academic health centers. Am Stat 70(2):187–194. https://doi.org/10.1080/00031305.2015.1077727
Mathews KS, Durst MS, Vargas-Torres C, Olson AD, Mazumdar M, Richardson LD (2018) Effect of emergency department and ICU occupancy on admission decisions and outcomes for critically ill patients. Crit Care Med 46(5):720–727. https://doi.org/10.1097/CCM.0000000000002993
Hiensch R, Poeran J, Saunders-Hao P, Adams V, Powell CA, Glasser A, Mazumdar M, Patel G (2017) Impact of an electronic Sepsis initiative on antibiotic use and health care facility-onset clostridium Difficile infection rates. Am J Infect Control 45(10):1091–1100. https://doi.org/10.1016/j.ajic.2017.04.005
Holzer H, Reisman A, Marqueen KE, Thomas AT, Yang A, Dunn AS, Jia R, Poeran J, Cho HJ (2019) "lipase only, please": reducing unnecessary amylase testing. Jt Comm J Qual Patient Saf 45(11):742–749. https://doi.org/10.1016/j.jcjq.2019.08.003
Shinwa M, Bossert A, Chen I, Cushing A, Dunn AS, Poeran J, Weinstein S, Cho HJ (2019) "THINK" before you order: multidisciplinary initiative to reduce unnecessary lab testing. J Healthc Qual 41(3):165–171. https://doi.org/10.1097/JHQ.0000000000000157
Cho HJ, Khalil S, Poeran J, Mazumdar M, Bravo N, Wallach F, Markoff B, Lee N, Dunn AS (2017) "lose the tube": a choosing wisely initiative to reduce catheter-associated urinary tract infections in hospitalist-led inpatient units. Am J Infect Control 45(3):333–335. https://doi.org/10.1016/j.ajic.2016.10.023
Tsega S, O'Connor M, Poeran J, Iberti C, Cho HJ (2018) Bedside assessment of the necessity of daily lab testing for patients nearing discharge. J Hosp Med 13(1):38–40. https://doi.org/10.12788/jhm.2869
Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai, the Mount Sinai Health System. http://icahn.mssm.edu/research/institute-health-care-delivery/. Accessed 4/30/2020
Blachman NL, Leipzig RM, Mazumdar M, Poeran J (2017) High-risk medications in hospitalized elderly adults: are we making it easy to do the wrong thing? J Am Geriatr Soc 65(3):603–607. https://doi.org/10.1111/jgs.14703
Howard DR, Kazemi N, Rubenstein WJ, Hartwell MJ, Poeran J, Chang AL, Podolnick JD, Parsons BO, Galatz LM, Flatow EL (2017) Cost-benefit analysis of routine pathology examination in primary shoulder Arthroplasty. J Shoulder Elb Surg 26(4):674–678. https://doi.org/10.1016/j.jse.2016.09.028
Poeran J, Babby J, Rasul R, Mazumdar M, Memtsoudis SG, Reich DL (2015) Tales from the wild west of US drug pricing: the case of intravenous acetaminophen. Reg Anesth Pain Med 40(3):284–286. https://doi.org/10.1097/AAP.0000000000000231
Duke Clinical Research Institute, Duke University Health System. 2018. Pragmatic Clinical Trials: Taking Novel Approaches to Answer Important Questions More Efficiently. https://dcri.org/insights/pragmatic-clinical-trials/. Accessed 4/30/2020
Brown CA, Lilford RJ (2006) The stepped wedge trial design: a systematic review. BMC Med Res Methodol 6:54. https://doi.org/10.1186/1471-2288-6-54
Galsky MD, Diefenbach M, Mohamed N, Baker C, Pokhriya S, Rogers J, Atreja A, Hu L, Tsao CK, Sfakianos J, Mehrazin R, Waingankar N, Oh WK, Mazumdar M, Ferket BS (2017) Web-based tool to facilitate shared decision making with regard to Neoadjuvant chemotherapy use in muscle-invasive bladder Cancer. JCO Clin Cancer Inform 1:1–12. https://doi.org/10.1200/CCI.17.00116
Timsina P, Kia A, Cheng FY, Parchure P, Otto W, Freeman R, Levin M. Real-time machine learning pipeline: a clinical early warning score (EWS) use-case. Paper presented at the HiMMS19 Champions of Health Unite. Global Conference & Exhibition., Orlando, FL, February 12, 2019
Zhong X, Lin JY, Li L, Barrett AM, Friedman J, Poeran J, Mazumdar M Derivation and validation of a delirium comorbidity index. In Press. Health Serv Res
Mazumdar M, Lin JJ, Zhang W, Li L, Liu M, Dharmarajan K, Sanderson M, Isola L, Hu L (2020) Comparison of statistical and machine learning models for healthcare cost data: a simulation study motivated by oncology care model (OCM) data. BMC Health Serv Res 20(1):350. https://doi.org/10.1186/s12913-020-05148-y
Kia A, Timsina P, Joshi HN, Klang E, Gupta RR, Freeman RM, Reich DL, Tomlinson MS, Dudley JT, Kohli-Seth R, Mazumdar M, Levin MA (2020) MEWS++: enhancing the prediction of clinical deterioration in admitted patients through a machine learning model. J Clin Med 9(2). https://doi.org/10.3390/jcm9020343
Timsina P, Cheng F, Joshi H, Kersch I, Wilson S, Colgan C, Freeman R, Mazumdar M, Levin M, Kia A (2020) MUST-Plus: A machine learning classifier that improves malnutrition screening in acute care facilities. J Am Coll Nutr. https://doi.org/10.1080/07315724.2020.1774821
Parchure P, Joshi H, Dharmarajan K, Freeman R, Reich DL, Mazumdar M, Timsina P, Kia A (2020) Development and validation of a machine learning-based prediction model for near-term in-hospital mortality among patients with COVID-19. BMJ Support Palliat Care. https://doi.org/10.1136/bmjspcare-2020-002602
Phelps, C. E. 2017. Health economics Boston: Addison Wesley
Capan M, Khojandi A, Denton BT, Williams KD, Ayer T, Chhatwal J, Kurt M, Lobo JM, Roberts MS, Zaric G, Zhang S, Schwartz JS (2017) From data to improved decisions: operations research in healthcare delivery. Med Decis Mak 37(8):849–859. https://doi.org/10.1177/0272989x17705636
Ferket BS, Feldman Z, Zhou J, Oei EH, Bierma-Zeinstra SM, Mazumdar M (2017) Impact of Total knee replacement practice: cost effectiveness analysis of data from the osteoarthritis initiative. BMJ 356:j1131. https://doi.org/10.1136/bmj.j1131
Giampaolo G, Ferket BS, D’Alessandro DA, Shi W, Horvath KA, Rosen A, Welsh S, Bagiella E, Neill AE, Williams DL (2016) Diabetes and the Association of Postoperative Hyperglycemia with clinical and economic outcomes in cardiac surgery. J Diabetes care 39(3):408–417. https://doi.org/10.2337/dc15-1817
Ferket BS, van Kempen BJ, Hunink MG, Agarwal I, Kavousi M, Franco OH, Steyerberg EW, Max W, Fleischmann KE (2014) Predictive value of updating Framingham risk scores with novel risk markers in the U.S. general population. PLoS One 9 (2). https://doi.org/10.1371/journal.pone.0088312
Shrime MG, Ferket BS, Scott DJ, Lee J, Barragan-Bradford D, Pollard T, Arabi YM, al-Dorzi HM, Baron RM, Hunink MGM, Celi LA, Lai PS (2016) Time-limited trials of intensive Care for Critically ill Patients with Cancer: how long is long enough? JAMA Oncol 2(1):76–83. https://doi.org/10.1001/jamaoncol.2015.3336
van Kempen BJ, Ferket BS, Steyerberg EW, Max W, Hunink MG, Fleischmann KE (2016) Comparing the cost-effectiveness of four novel risk markers for screening asymptomatic individuals to prevent cardiovascular disease (CVD) in the US population. Int J Cardiol 203:422–431. https://doi.org/10.1016/j.ijcard.2015.10.171
Xia FD, Ferket BS, Huang V, Stern RS, Wu PA (2019) Local radiation and phototherapy are the Most cost-effective treatments for stage IA mycosis Fungoides: a comparative decision analysis model in the United States. J Am Acad Dermatol 80(2):485–492. https://doi.org/10.1016/j.jaad.2018.07.040
Wasserman I, Poeran J, Zubizarreta N, Babby J, Serban S, Goldberg AT, Greenstein AJ, Memtsoudis SG, Mazumdar M, Leibowitz AB (2018) Impact of intravenous acetaminophen on perioperative opioid utilization and outcomes in open colectomies: a claims database analysis. Anesthesiology 129(1):77–88. https://doi.org/10.1097/ALN.0000000000002227
The Center for Qualitative Studies in Health Medicine. Johns Hopkins Bloomberg School of Public Health. 2018. https://jhsph.edu/departments/health-behavior-and-society/research-and-centers/center-for-qualitative-studies-in-health-and-medicine/index.html. Accessed 4/30/2020
Gorbenko KO, Fraze T, Lewis VA (2016) Redesigning care delivery with patient support personnel: learning from accountable care organizations. Int J Care Coord 19(3–4):73–83. https://doi.org/10.1177/2053434516676080
Schwartz RM, Gorbenko K, Kerath SM, Flores R, Ross S, Taylor TN, Taioli E, Henschke C (2018) Thoracic surgeon and patient focus groups on decision-making in early-stage lung Cancer surgery. Future Oncol 14(2):151–163. https://doi.org/10.2217/fon-2017-0254
Patients Like Me. https://www.patentslikeme.com/. Accessed 4/30/2020
Brooks JV, Gorbenko K, Bosk C (2017) Interactional resources for quality improvement: learning from participants through a qualitative study. Qual Manag Health Care 26(2):55–62. https://doi.org/10.1097/QMH.0000000000000128
Mazumdar, M. Supporting reproducibility in a learning health system: surveying clinical data quality-related practices, assessing awareness and perspectives, and gauging training needs. In National Academy of Sciences Arthur M. Sackler Colloquium on Reproducibility of Research: Issues and Proposed Remedies. Washington, D.C.
Khairat S, Whitt S, Craven CK, Pak Y, Shyu CR, Gong Y (2019) Investigating the impact of intensive care unit interruptions on patient safety events and electronic health records use: an observational study. J Patient Saf. https://doi.org/10.1097/PTS.0000000000000603
Gorbenko K, Mendelev E, Keefer L (2019) Can multidisciplinary team meetings reduce burnout? J Eval Clin Pract 26:1–3. https://doi.org/10.1111/jep.13234
Marqueen KE, Waingankar N, Sfakianos JP, Mehrazin R, Niglio SA, Audenet F, Jia R, Mazumdar M, Ferket BS, Galsky MD (2018) Early mortality in patients with muscle-invasive bladder Cancer undergoing cystectomy in the United States. JNCI Cancer Spectr 2(4):1–8. https://doi.org/10.1093/jncics/pky075
Budde J, Agarwal P, Mazumdar M, Yeo J, Braman SS (2018) Can an emergency department observation unit reduce hospital admissions for COPD exacerbation? Lung 196(3):267–270. https://doi.org/10.1007/s00408-018-0102-1
Ferket BS, Ailawadi G, Gelijns AC, Acker M, Hohmann SF, Chang HL, Bouchard D et al (2018) Cost-effectiveness of mitral valve repair versus replacement for severe ischemic mitral regurgitation: a randomized clinical trial from the cardiothoracic surgical trials network. Circ Cardiovasc Qual Outcomes 11(11):1–26. https://doi.org/10.1161/CIRCOUTCOMES.117.004466
Mazumdar M, Messinger S, Finkelstein DM, Goldberg JD, Lindsell CJ, Morton SC, Pollock BH, Rahbar MH, Welty LJ, Parker RA, Biostatistics, Epidemiology, and Research Design (BERD) Key Function Committee of the Clinical and Translational Science Awards (CTSA) Consortium (2015) Evaluating academic scientists collaborating in team-based research: a proposed framework. Acad Med 90(10):1302–1308. https://doi.org/10.1097/ACM.0000000000000759
Gourevitch MN, Thorpe LE (2019) Advancing population health at academic medical centers: a case study and framework for an emerging field. Acad Med 94(6):813–818. https://doi.org/10.1097/acm.0000000000002561
Pronovost PJ, Holzmueller CG, Molello NE, Paine L, Winner L, Marsteller JA, Berenholtz SM, Aboumatar HJ, Demski R, Armstrong CM (2015) The Armstrong institute: an academic Institute for Patient Safety and Quality Improvement, research, training, and practice. Acad Med 90(10):1331–1339. https://doi.org/10.1097/acm.0000000000000760
Costlow MR, Landsittel DP, James AE 3rd, Kahn JM, Morton SC (2015) Model for a patient-centered comparative effectiveness research center. Clin Transl Sci 8(2):155–159. https://doi.org/10.1111/cts.12257
Rask KJ, Brigham KL, Johns MM (2011) Integrating comparative effectiveness research programs into predictive health: a unique role for academic health centers. Acad Med 86(6):718–723. https://doi.org/10.1097/ACM.0b013e318217ea6c
Rosenblum D (2012) Access to Core facilities and other research resources provided by the clinical and translational science awards. Clin Transl Sci 5(1):78–82. https://doi.org/10.1111/j.1752-8062.2011.00385.x
Mazumdar M, Moshier EL, Ozbek U, Parsons R (2018) Ten essential practices for developing or reforming a biostatistics Core for a NCI designated Cancer center. JNCI Cancer Spectr 2(1):1–6. https://doi.org/10.1093/jncics/pky010
Acknowledgements
We acknowledge Jeremy Boal, MD, President of Mount Sinai Downtown and Executive Vice President and Chief Clinical Officer of the Mount Sinai Health System. He was the Associate Director of the Institute for its first three years and was instrumental in its initiation and growth. We also recognize the I-HDS Internal Advisory Board (IAB), which comprises Dean Dennis S. Charney and the Presidents, Chief Medical Officers, and Chief Nursing Officers from the Mount Sinai Health System’s seven hospitals, and the chairs/vice chairs/directors of all collaborating departments and Institutes. We thank our External Advisory Board: Peter R. Orszag, PhD, Vice Chair, Corporate and Investment Banking, Citigroup, Inc.; Patricia Wang, JD, Chief Executive Officer, Healthfirst, Inc.; Isaac Kohane, MD, PhD, Co-Director, Center for Bioinformatics, Henderson Professor of Pediatrics and Health Sciences and Technology, Harvard Medical School; and Gregory L. Burke, MD, MSc, Senior Associate Dean for Research and Chief Scientific Officer, Wake Forest Baptist Health. We acknowledge all members of the I-HDS: Umut Ozbek, PhD, Denise Williams, BS, Jiayi Ji, MS, Lihua Li, MS, PhD, Erin Moshier, MS, Liangyuan Hu, PhD, Rachel Jia, PhD, Jung-Yi (Joyce) Lin, MS, Meng Ru, MS, Xiaoyu Song, PhD, Hsin-Hui (Vivien) Huang, MS, PhD, MD, Himanshu Joshi, MBBS, PhD, and Serene Zhan, MS. Finally, we thank our collaborating clinical departments, centers, and institutes for their support.
This work was supported in part by the National Cancer Institute (P30 CA196521);
The National Heart, Lung, and Blood Institute (U01 HL088942); and the National Institute of Aging (P30 AG028741 and P30 AG028741- 09S1).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mazumdar, M., Poeran, J.V., Ferket, B.S. et al. Developing an Institute for Health Care Delivery Science: successes, challenges, and solutions in the first five years. Health Care Manag Sci 24, 234–243 (2021). https://doi.org/10.1007/s10729-020-09521-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10729-020-09521-5