INTRODUCTION

Clinical research is vital to ensure continuing advances in health care. However, in the United States as well as in Europe, medical education provides only limited training in clinical research, and a decline in the number of physician-scientists has been identified for at least 2 decades.1–7 Therefore, the National Institute of Health has recently made substantial investments in training programs and career development awards for junior investigators.3,8 A wide range of clinical research programs is now accessible,9–11 including feasible types of scaled-down programs in settings where funding is not available.

A systematic review of resident research curricula showed some encouraging results regarding increased research productivity of the participants.12 However, in the 41 studies reviewed, evaluation methods were often rudimentary, only 5 studies reported pre-post intervention testing of learners’ knowledge, and no curriculum was evaluated as a prospective controlled study.12

A recent study in the German health care system confirmed that subjective research skills, objective research knowledge, and research productivity are relatively low among postgraduate physicians and psychologists.13 To improve this situation, a resident training program in clinical research was launched at the University Medical Center Heidelberg in 2005. It was established during residency to allow an early initiation into research training and to better integrate clinical and research training years.14 Given that clinical research requires expertise of many kind of investigators,8 the training program included both physicians and psychologists. A 1-year period was chosen to provide rigorous research training without excessive length.2

This study investigates the effectiveness of the structured 1-year resident training program in clinical research using a controlled before-and-after study design. In addition to the primary outcomes of research knowledge and productivity, participant evaluation of the training program was assessed.

METHODS

Subjects and Study Design

A 1-year training program in clinical research was conducted in the Department of Psychosomatic and General Internal Medicine at the University of Heidelberg from October 2005 to October 2006. To assess its effectiveness, we compared program participants to a control group from 2 similar university departments of Psychosomatic Medicine at the University of Tübingen and at the Technical University of Munich. To achieve comparability among the study centers, we chose 3 Medical Schools with top rankings in research and teaching.15,16 In the largest German university ranking, including all 35 German Medical Schools, the medical faculties of Heidelberg, Munich, and Tübingen ranks #2, #1, and #6, respectively.17 The medical schools of Heidelberg, Munich, and Tübingen are also similar in terms of number of students in the clinical section of their medical education (1,586; 1,337, and 1,522, respectively), the number of full professors for the clinical specialties (75, 76, and 72, respectively), annual research money per full professor (438,000€, 458,000€, and 484,000€, respectively), and annual publications per full professor (19.8, 17.5, and 22.1, respectively).15 The study subjects included residents in Internal Medicine, psychotherapy and psychosomatics, psychiatry, and psychology. Residencies for both physicians and psychologists in Germany include internship and have a total duration of 5 to 6 years. The career paths for residents at the 3 participating universities are identical and all 3 departments have a main focus in clinical research. Residents at university departments in Germany are expected to perform research, but the high priority for patient care poses some constraints on research productivity.

At the time of this study, structured training programs for clinical research were not provided at either of the control sites. At all 3 sites, inclusion criteria were identical: residency in medicine or psychology, active participation in a research project, and informed consent. Exclusion criteria were insufficient knowledge of the German language. Demographic characteristics and study outcomes were assessed at the beginning of the program (baseline) and 1 week after the end of the program (1-year follow-up). Participation in the program was free of charge. All residents from the Department of Psychosomatic and General Internal Medicine at the University of Heidelberg working in a research project were expected to participate in the training program and to complete both assessments. Correspondingly, all residents working in a research project in the corresponding departments in Munich and Tübingen were expected to complete both assessments.

Training Program in Clinical Research

The training program consisted of 3 elements shown to be associated with successful research activity:12 a) provision of methodological research knowledge within the scope of a “Clinical Research Methods” course, b) mentorship by an experienced researcher, and c) work on an individual research project. The “Clinical Research Methods” course was conducted once a week with a total of 33 lessons lasting 90 minutes each. A total of 20 lecturers held the lessons. Structure and content of the course are detailed in Table 1. The mentors were encouraged to follow rules for effective mentorship,18,19 but no additional time for mentoring was available. The training program was not supported by extramural funding and was conducted as “in service training”. The participants had no extra time for research. The structure of the Heidelberg curriculum is comparable to other training programs in clinical research, such as the “Clinical Investigator Training Enhancement (CITE) Program” of the Regenstrief Institute, Indiana University, Indianapolis (http://www.regenstrief.org/training/research), but the dose of the individual training elements is considerably lower because of the necessary integration into a busy work setting.

Table 1 Clinical Research Methods Course
Full size table

Measures

We used several different assessment methods to cover the different dimensions of research knowledge and research productivity and to compensate for potential limitations in any 1 method.20 Methodological research knowledge was considered the primary outcome because knowledge of a domain has proven to be the single best determinant of expertise.21,22 Cognitive knowledge is best assessed using written form tests, and multiple-choice questions have especially high reliability.21–24 Therefore, we used multiple-choice questions to assess methodological research knowledge according to Miller’s pyramid of competence.25 The multiple choice items with 3 different question formats, namely, single best answer, pick n item format, and K-PRIM type, were constructed by the lecturers of the “Clinical Research Methods” course in accordance with multiple-choice question item-writing principles.26,27 Pretesting ensured the quality of the multiple-choice questions, and a panel of 4 experts reviewed the items. A total of 28 questions covering “biostatistical knowledge” (15 items) and “basics of clinical research” (13 items) fulfilled the quality standards and were included in the multiple-choice test. To evaluate participants’ learning progress, the test was written twice in identical format according to progress-testing principles.28 Internal consistency, assessed at follow-up, was good for the total scale (α = 0.84) and acceptable for the 2 subscales (biostatistical knowledge, α = 0.81; basics of clinical research, α = 0.60).

Self-assessed research competence was measured with 12 items covering “interpretation of clinical studies”, “designing clinical studies”, “biostatistical competence”, and “presentation of study results” (3 items each, e.g., item 7: “I feel competent in writing original journal articles”). Six-point Likert scales were used to assess agreement with the items with higher scores indicating higher self-assessed research competence. Internal consistency of the total self-assessed research competence scale (α = 0.95) and the 4 subscales (α = 0.93, α = 0.82, α = 0.91, α = 0.85, respectively) was very good.

The number of original publications, reviews or meta-analysis, book articles, and grant proposals was assessed at the beginning and at the end of the program, differentiating between first- and co-authorship. An independent literature search on all participants, using the databases PUBMED, Web of Science, SCOPUS, PSYNDEX, and PSYCHLIT, confirmed that the subjects’ self-reports were valid. Given that the 1-year period of the training program was considered too short to expect a substantial effect in terms of published articles, we additionally assessed whether subjects had presented research results at scientific meetings during the training program and whether they were currently writing original journal articles and book articles at the end of the program.

The quality of the program was investigated by an evaluation of each of the individual 33 “Clinical Research Methods” lessons directly after the course using 5 questions regarding relevance of content, didactic quality, commitment of the lecturer, increase of knowledge, and overall evaluation of the lesson. The quality of the whole training program was assessed at the end of the program using 5 questions regarding quality of content, lecturers, teaching methodology, utility for one’s own career development, and relevance to one’s own research activity. Six-point Likert sales were used for those purposes. Participants were also asked if they would participate again in a similar program. Finally, a focus group was conducted at the end of the program to more thoroughly assess the participants’ perceptions of the program and potential opportunities for improvement.29,30

Statistical Analysis

Program effectiveness was tested by comparing change scores of the continuous outcomes using Analysis of Covariance (ANCOVA) according to the General Linear Model, controlling for the potential confounders of gender, age, and dissertation status. Change scores were analyzed because this allows progress testing28,31 independent from the level at baseline and because this method is especially efficient if baseline and follow-up data are highly intercorrelated.32,33 For comparisons of the number of publications and grant proposals, the statistical assumptions for analysis of variance were not met because the number was generally low and the distributions were skewed. Therefore, we performed χ2 tests or Fisher’s exact tests (if cell sizes <5) for these outcomes.

Given that the sample size in the intervention group was fixed by the size of the class of the training program, power analysis was performed for an estimated sample of 15 intervention subjects and 21 control subjects at the end of the program. Given a 0.05 level of significance (two-sided) and a power of 0.80, this sample size was sufficient to detect large differences of approximately 1.0 standard deviation between the groups.34 Group differences between continuous outcomes were also transformed to effect sizes (difference between mean scores divided by standard deviation of control group).35 Missing values were extremely rare (<1%). Analyses included data only for subjects who completed baseline and follow-up assessments. Imputations for loss to follow-up using the last observation carried forward did not significantly change the results. To minimize bias, all multiple-choice questions were evaluated using a computer program. Statistical analyses were performed using Statistical Analysis System (SAS, Version 9.1; Cary, NC, USA).

RESULTS

Characteristics of Intervention and Control Subjects

Of the 20 residents who registered for the training program, 2 were not admitted because they were not able to regularly participate in the clinical methods class. Eighteen subjects met inclusion criteria and started with the training program. During the course of the program, 2 subjects were not able to continue because of change of their professional employment, and 1 subject quit because of time conflicts. Thus, the 1-year training program was completed by 15 of 18 subjects (83%). All intervention subjects participated in the baseline and follow-up assessments. In the control group, baseline participation rate was 81% (26 of 32) and follow-up rate was 85% (22 of 26). Table 2 shows baseline demographic characteristics, methodological research knowledge, self-assessed research competence, publications, and grant proposals of subjects who completed both baseline and follow-up assessments. The groups did not differ significantly with respect to age, years of research experience, university degree, dissertation status, self-assessed research competence, written publications, and grant applications. However, there were fewer women in the intervention compared to the control group, and intervention subjects achieved higher baseline test scores regarding research knowledge.

Table 2 Baseline Characteristics of Study Subjects and Controls at Baseline
Full size table

Change in Methodological Research Knowledge and Self-assessed Research Competence

Table 3 shows the change in methodological research knowledge and change in self-assessed research competence for intervention subjects and controls. For both outcomes, intervention subjects experienced greater improvement than did controls. As expected, baseline and follow-up scores were highly intercorrelated, both of methodological research knowledge (total score, r = .58; subscale scores, r = .84, and r = 62; all p < .001) and self-assessed research competence (total score, r = .78; subscale scores, r = .72, r = .81, r = .43, r = .83, resp.; all p < .01). This supported the statistical comparisons of change scores.32 The methodological research knowledge total score improved by a mean of 8.4 (SD = 2.9) additional correct answers in the intervention group compared to 1.2 (SD = 2.8) additional correct answers in the control group (effect size = 2.5; p < .001). With respect to self-assessed research competence, intervention subjects had significantly greater improvements in the total scale score as well as the subscales “interpretation of clinical studies” and “biostatistical competence”.

Table 3 Change in Methodological Research Knowledge and Self-Assessed Research Competence After 1 Year
Full size table

Publications and Grant Proposals During Time of Training Program

Outcomes regarding actual research activity during the year of the training program are summarized in Table 4. Significantly more intervention subjects compared to controls were currently working on a journal article (87% vs 36%, p = .003), had presented scientific results at a research meeting (80% vs 41%, p = .04), and had completed at least 1 original paper as co-author (60% vs 18%, p = .01) during the time of the training program. However, the groups did not differ significantly with respect to completed original articles as first author, reviews, meta-analyses, or book articles. With respect to grant applications, intervention subjects had completed significantly more applications for funding as co-investigators compared to controls (47% vs 5%, p = .004), but not as principal investigators. Finally, intervention subjects had significantly more grant applications accepted for funding (33% vs 0%, p = .007).

Table 4 Publications and Grant Proposals During 1 Year of Training Program
Full size table

Evaluation of Training Program

The 33 individual sessions of the “Clinical Research Methods” course received an excellent evaluation with a mean evaluation score between “very good” and “good” (Table 5). Similarly, the training program as a whole received high evaluation scores. Fourteen of the 15 participants (93%) indicated they would participate again in a similar program.

Table 5 Evaluation of Clinical Research Course by Intervention Group
Full size table

Content analysis of the focus group, in which 13 of the 15 (87%) intervention subjects participated, revealed that the participants evaluated the training program as profound and highly valuable for becoming an independent researcher as well as for the own career development. As an additional benefit, the participants noted that the program generated a positive research culture and enhanced research as a priority alongside clinical work and teaching. With respect to mentorship, the intervention subjects noted that contact time with the mentor was generally low and that it was not higher compared to the time before the program. In addition, the participants felt somewhat overloaded with the time requirements for the program in that their clinical workload was not reduced.

DISCUSSION

This study has several major findings. First, the structured 1-year training program led not only to improved research knowledge and self-assessed research competence; it was also accompanied by increased research productivity. Second, enhanced research activity consisted of more presentations at scientific meetings, increased writing activity, and co-authorship on original papers and grant applications. This suggests that, whereas the intervention subjects made great progress, 1 year was, as expected, insufficient to produce independent investigators. Third, the integration of such a training program into residencies with demanding clinical workloads is feasible. Despite competing demands, the participating residents highly valued the training program. Our controlled before-and-after study design substantiates the effectiveness of similar training programs from earlier studies using cross-sectional or pre-post uncontrolled intervention designs.12 Given that residents in medical specialties have passed through the same medical education system and given that an earlier study did not find differences in research output between physicians and psychologists,13 it is likely that our results also apply to other medical specialties and health care professions.

With the provision of methodological research knowledge, mentorship by an experienced researcher, and work on an individual research project, the program included 3 elements established as key factors for successful research training.12 Nevertheless, owing to the lack of extramural funding, other important factors such as protected time for research and extra funding for travel costs to scientific meetings36,37 were not provided. As evidenced by the focus group at the end of the training program, the most frequently encountered obstacle was lack of time. In fact, insufficient time is probably the most common obstacle to completing research in general.38,39 Even without protected time, however, the program was moderately effective. In particular, this type of program may be more generalizable to institutions not having substantial federal support for research training. Suggestions for maximizing limited research time39 that were provided at the beginning of the program might have helped the participants to structure their time for completing their research projects. Most importantly, the program made research a priority, generated a positive research culture, helped in understanding strengths and weaknesses of clinical research studies, and encouraged networking among the participants. In addition to the structured provision of research knowledge, these factors might have motivated the participants of the training program to successfully complete their research projects.

Our study has several limitations. First, our sample size was limited by the size of the training program. Second, the study was controlled but participants were not randomized. Whereas we were able to adjust for some group differences, the possibility of unmeasured confounders remains. Third, the intervention group had somewhat higher scores in the multiple-choice test regarding methodological research knowledge at the beginning of the program. However, our outcome was change in methodological research knowledge,32,33 and the increase of research knowledge was much greater in the intervention subjects compared to the controls. Finally, whereas the knowledge provided in the Clinical Research Class was identical for all participants, the intensity of mentorship varied depending on the individual mentor–trainee dyad. However, given that the focus group revealed that the time with the mentor was not substantially increased by the program, differences in mentorship between the study sites are unlikely to be a major factor accounting for the study findings.

In summary, a comprehensive 1-year resident training program can increase research knowledge, self-assessed research competence, and research productivity. Such a program might be advantageous for other academic institutions with minimal external funding for clinical research training. Nevertheless, insufficient time appears as a key obstacle to completing research projects38 and taking on larger roles as first authors and principal investigators. Therefore, obtaining funding for at least some research time should remain a priority. In the meanwhile, structured training programs in clinical research can be beneficial despite resource constraints.