INTRODUCTION

There is increasing awareness of the importance of critically evaluating research investments [1–4] and whether their returns reflect the goals of the funding agencies [5]. Most funding agencies are supported by taxpayers (governmental agencies) or by philanthropic donations [1]. Both researchers and their respective funding agencies should provide evidence that their funds are, in fact, being used to promote new findings related to health and wellness appropriately. The successful role of funding agencies in promoting knowledge related to health and medicine is evident in cancer research [6].

Some small grant funding agencies are committed to supporting novice investigators. The scholarly and scientific productivity of junior and novice investigators in healthcare and the factors associated with sustained productivity are ongoing topics of discussion in academic circles [7, 8], New investigators appear to benefit from research funding opportunities early in their careers [7, 9]. However, there are questionable benefits in supporting junior faculty without investigating their scientific productivity and future research initiatives.

The Hamilton Health Sciences (HHS) in affiliation with McMaster University is an academic organization that fosters education, learning, and research through a number of foundations and hospitals. The HHS Foundation is a research-intense environment. It promotes collaboration and growth, giving researchers an opportunity to advance their research interests and collaborate with other departments and institutes. Research at HHS is conducted through a series of research institutes, research centers and groups, as well as investigators who work independently. HHS supports the growth of trainees, research fellows, and scholars. HHS fosters research growth and program development through a series of internal funding award programs for all healthcare professionals. HHS commitment to “first stage” research led to the creation of the New Investigator Fund (NIF) in 2004.

NIF is a non-profit funding organization that nurtures the research careers of novice investigators and trainees while supporting research initiatives relevant to the clinical mission and strategic research directions of HHS. The fund provides a unique opportunity to foster a culture of inquiry for novice investigators with the support and mentorship of senior staff. The NIF provides research funding to front-line staff–physicians and other healthcare professionals–using a comprehensive peer-reviewed process. Priorities are given to grant proposals that (i) are consistent with, and likely to enhance, the main clinical programs at HHS; (ii) are multidisciplinary and collaborative; (iii) build on existing research strengths; (iv) advance and create new knowledge to inform patient care providers by bringing evidence into practice through translation and application; (v) conduct pilot or feasibility studies which have the potential to facilitate funding of full-scale trials (www.hhsresearchadmin.ca/researcher-support/internal-funding/new-investigator-fund/).

This report evaluates the performance of NIF since its inception in 2004. We used a mixed method approach to examine if award recipients established better scientific productivity compared to non-recipients. The primary objective was to compare the peer-reviewed publication of the NIF-submitted application as a measure of scientific productivity between awardees and non-awardees. The secondary objectives were to compare the journal impact and the citations of the NIF-published article. We also qualitatively examined the applicants’ perspectives, experiences, successes, and challenges on the NIF grant application process and its impact on their scientific productivity and career.

METHODS

Design

This is mixed methods design of quantitative and qualitative data. The NIF electronic database of the submitted grant applications and follow-up surveys were retrospectively reviewed from inception in 2004 to the end of 2013. We initiated this report based on a predefined protocol and with the approval from HHS and McMaster University Research Ethics Board in 2012.

Applicant eligibility

Eligible applicants were novice investigators within the first 5 years of conducting independent research with HHS affiliation. Applicants included physicians, nurses, research and clinical fellows, clinical scholars (clinicians or health professionals received awards to advance their clinical knowledge), residents, health professionals (pharmacologists/pharmacists, therapists, epidemiologists, kinesiologists), or graduate and undergraduate students conducting research in HHS. The discipline and expertise, affiliation, level of education, and gender of the applicants and the type or topic of application did not limit eligibility.

NIF grant review process

The NIF Scientific Peer-Review Board comprised of mid- to senior-career faculty members from different disciplines with a wide scope of research and clinical knowledge and expertise. Members of the “Committee” were primary and secondary reviewers of the applications. NIF held two competitions annually with submission deadlines of March 31 and October 1 with a permissible maximum budget of $50,000 per application. The total amount of available funds was consistent over the years. The Research Administration Office screened the electronically submitted applications for eligibility and then assigned them to a primary and a secondary reviewer. The reviewers ranked three aspect of the applications: (1) eligibility of applicant—research experience irrespective of publications and whether they had research education such as degree, certificate, or workshops (curriculum vitae (CV)); (2) scientific merits of the research proposal for feasibility and novelty of the research question, design and methods, and timeline; (3) credibility of mentor—research education and productivity and research facility (CV); and (4) budget and expenditure justification. Each reviewer summarized strengths and weaknesses of the applications. The Board Members then met, discussed the ranked applications, and the corresponding comments on the proposed methodology. The Board Members then made decisions on acceptance, conditional acceptance, resubmission, or rejection of the applications. The applicant’s education, expertise and discipline, and university affiliation did not affect the funding decisions. The requested funding, as long as it was within the permissible amount of $50,000, also did not limit the decision. Conditional acceptance was decided when the application required some methodological or budgetary adjustments or clarifications. Resubmission was recommended when the application had scientific merits but required methodological revision. We asked the applicants to revise the proposal and response to the reviewers’ comments and resubmit the proposal. The acceptance or rejection of applications was based on the final submission with improved and acceptable methodology.

The Research Administration generated a letter to communicate the Board’s decision with respective anonymous reviewers’ comments to each applicant. The Research Administration monitored the progress of the award recipients annually until completion of the NIF applications via a short electronic survey. We grouped the applicants as awardees and non-awardees of NIF funding and analyzed the data accordingly.

Data sources

To avoid duplicate entries, we included the applications based on the final decision. The applicants’ name, gender, email address, position, education, affiliation, and discipline/expertise were collected from the application form and CVs. The application details were collected from the submitted research proposal. To increase power, we combined some of the expertise into one category based on their relevancy due the small numbers.

The information on the progress of the applications and perceptions, successes, and challenges of the applicants (using open-ended questions) collected by the Research Administrative Office was available in the online NIF database. No data was collected on the non-recipients by this office. Survey data was not collected on the non-recipients. The authors sent the same survey questionnaire to the non-recipients using the contact information provided in the original application in January 2013. Some emails were not delivered due to discontinued accounts. To locate applicants with undeliverable address, we first searched the HHS and McMaster University staff directory and then searched the Internet matching for demographics. We sent multiple reminders to non-responders to optimize the response rate. In failing to locate an applicant or receive response, we contacted the application mentor to collect the information with respect to the progress of the submitted application. We updated the survey data on the uncompleted applications on March 2015 and August 2015.

The short survey questionnaire included the following questions:

  1. 1.

    Were you awarded NIF funding? If not, did you receive funding from elsewhere? (yes, no)

  2. 2.

    What is the current status of the application? (completed, in progress or discontinued) If discontinued, please provide us with the reasons.

  3. 3.

    Was any manuscript submitted or published?

  4. 4.

    Was any further research produced from this application? If yes, was the funding secured? Was it a randomized controlled trial?

The open-ended questions were also included to collect the qualitative data on the successes and challenges of the applicants with respect to the NIF submitted application. The open-ended questions were independently classified in different themes by two reviewers and disagreements were resolved with consensus:

  1. 1.

    What was your greatest success in pursuing this application?

  2. 2.

    What was your greatest challenge in pursuing this application?

  3. 3.

    Did your application inspire any other ideas?

  4. 4.

    If you were denied the NIF award, were you discouraged to reapply to NIF or other funding agencies?

Outcome measures

The publication of the NIF-submitted application was our primary outcome measure. The impact factor of the journal and the number of citations of published NIF application were the secondary outcome measures.

Primary and secondary outcome measures

The search for the publications of the NIF application was conducted in December 2014 and updated in March and August 2015 to include the most recent publications. To minimize bias and maximize accuracy, two reviewers independently searched the online databases. The disagreements were resolved with consensus. Reviewers first searched the full name of the applicant in Medline and Pubmed. If the search produced substantial and heterogeneous citations, it was narrowed down by adding the search term of “AND (Hamilton or McMaster).” Reviewers then read the abstracts and the full text if necessary to confirm the publication of the NIF application. They recorded the article identifier number and name of the journal. The impact factor of the journal was retrieved from the Journal Citation Reports (2012 JCR Science Edition). The impact factor was unattainable for six journals. The number of citations of each article was obtained from the Science Citation Index Expanded™ via Web of Science and Google Scholar. The Kappa statistics for the level of agreement on the publication of NIF-submitted application was 88.0 % (95 % CI 81.6–94.3 %).

Qualitative assessment of the survey data

We classified the open-ended questions into different themes for awardees and non-awardees using the modified themes from Mavis et al. [7]. Two reviewers independently reviewed the comments and categorized them into the classified themes. Disagreements were resolved with consensus.

Data analysis

We described the applicants’ and applications’ characteristics between awardees and non-awardees. We reported the categorical variables as frequencies and relative frequencies and compared using Chi-squared or Fisher’s exact tests. We reported the continuous variables as mean with standard deviation (SD) or medians with minimum and maximum values and compared them using unpaired t test or Mann-Whitney U test. We plotted bar charts for the visual presentation. We calculated odds ratios (OR) to quantify the effect of the NIF award on the publication of the submitted application and adjusted for applicants’ sex, education, discipline and affiliation, and applications’ design using binary logistic regression analysis. We checked for interactions between the variables. We reported ORs with 95 % confidence intervals (CI) and p value for Hosmer-Lemeshow goodness-of-fit test. A p value of 0.05 was considered for statistical significance. SPSS version 22.0 was used for statistical analysis (IBM, Chicago, IL).

RESULTS

A total of 296 applicants were entered into the competition between 2004 and 2013. Of the 296 applicants, 163 (55 %) were awarded funding. Of the 133 non-awardees, seven applicants were ineligible (not novice or unaffiliated with HHS). The other 126 non-awardees were declined for the following reasons: (i) the research question was not novel or feasible and (ii) the design and/or methodology were not sound. Of the 133 non-awardees, 24 (18.0 %) received awards from other sources. The follow-up survey on the application progress was completed by 268 (90.5 %) applicants. The 28 non-responders were non-awardees from earlier years (2004–2006). The mean requested funding was $29,950 (SD = $12,123) for awardees and $27,845 (SD = $15,588) for non-awardees (p = 0.014). The mean awarded funding varied from $21,800 to $34,770 from 2004 to 2013.

The awarding pattern did not significantly change over the years (p = 0.434) (Fig. 1), but there were some discrepancies. More applications were funded in the years 2004–2007, as smaller funds were requested [mean = $21,600 (SD = $5400)]. Fewer applications were submitted during the years 2010–2013 and more were awarded with a mean funding of $35,270 (SD = $9330). The number of the submissions picked up after 2012 but fewer applications were funded because larger funds were requested [mean = $34,800 (SD = $9100)].

Fig. 1
figure 1

Trend of awarding pattern 2004–2013 between NIF awardees and non-awardees

Full size image

Applicants’ and applications’ characteristics

Table 1 presents the baseline characteristics by awarding status. Applicants’ affiliation (p = 0.189), field of discipline and expertise (p = 0.123), and gender (p = 0.412) did not significantly differ between the awardees and non-awardees. Physicians with a graduate degree (30.7 %) and applicants with a Doctor of Philosophy (PhD) received more awards than physicians without graduate education and those with Masters of Science (MSc) and Bachelors of Science (BSc) degrees (p = 0.037). As for the study design, basic science research (28.8 %), RCTs (24.5 %), and pilot and feasibility trials (12.3 %) were awarded more often than rejected (p < 0.001). For observational designs, the applications (64.6 %) were more frequently rejected than awarded (34.4 %).

Table 1 Comparison of applicants’ and applications characteristics by between NIF awardees and non-awardees
Full size table

Scientific productivity of awardees and non-awardees

Table 2 presents the results of the multivariable logistic regression analysis. Of 114 applicants who published the NIF-submitted application, 85 (74.6 %) were awardees and 29 (25.4 %) non-awardees. Adjusting for the applicants and applications characteristics, awardees were 3.4 (95 % CI: 1.9, 5.9) times more likely to publish their NIF-submitted application than non-awardees (Hosmer-Lemeshow goodness-of-fit test p value = 0.755). None of the other factors significantly contributed to the publication of the NIF application except that applicants with MSc or BSc degrees published less often than other applicants (p = 0.038). The mean journal impact factor (3.7 (SD = 2.7) vs. 3.2 (SD = 2.1), p = 0.406), and the median NIF article citation (2 (0, 69) vs. 2 (0, 35), p = 0.988) did not differ between awardees and non-awardees, respectively (Table 3).

Table 2 Multivariable regression to examine the effect of NIF award on publication of the NIF-submitted applications
Full size table
Table 3 Journal impact factor and citation of NIF application published article by award status
Full size table

Of the 182 unpublished applications, 39 were discontinued (12 awardees and 27 non-awardees). Of the 12 awardees, five left HHS, five discontinued due to feasibility issues (slow recruitment and lack of sufficient data) and two did not comment on the reason. Of the 27 non-awardees, six applicants left HHS, nine stopped due to feasibility issues, and 16 failed to secure or seek funding. Of the remaining 143 unpublished applications, 39 submitted in 2012–2013 and 1 RCT awarded in 2010 were still ongoing.

Applicants’ opinion of the NIF award

Of 286 completed surveys, 166 (127 awardees; 39 non-awardees) answered the open-ended questions and specifically commented on the NIF award.

The comments from 127 awardees were classified in the following themes:

  • Influenced and established the direction of my research plans (84); generated further research (36)

  • Generated full-scale RCTs with secured funding (5)

  • Secured funding for further research (26)

  • Gained internal and external professional recognition and collaboration (23)

  • Transferred findings to relevant policy makers (two applicants) or supported health professionals’ curriculum development (one applicant)

  • Patented the findings (one applicant); won research award (one applicant)

  • Discouraged and discontinued the application due to feasibility issues (5).

The comments of 39 non-awardees revealed the following themes:

  • Gained sense of satisfaction by securing funding from other sources (24)

  • Managed to continue without funding (22)

  • Generated further research (4); initiated full-scale RCT with secured funding (1)

  • Gained professional recognition and collaboration (6)

  • Gained experience in writing a grant proposal and learning the application process (2); made me realize I lack research background and need to try harder (1)

  • Discouraged (16): did not seek funding (5); lost interest due to feasibility issues (9); found writing a grant application is challenging (2)

There were some interesting comments from non-recipients when we asked if they were discouraged as a result of the decision on their NIF application. One applicant said, “Yes. I was rejected from 3 funding sources for this project, and it is a lot of work to do the applications. I felt that if I self-funded a small portion then I could use the pilot data to demonstrate feasibility. That might increase my chances of getting funding for the project in the future.” Another applicant said, “Yes, non-physicians are not easily funded.” Two applicants believed that there was not enough funding for their field of specialty (pediatrics and hepatobiliary).

DISCUSSION

We evaluated the impact of the small NIF award on the scientific productivity of novice investigators. We primarily defined scientific productivity as publication of the NIF-submitted application and secondarily as production of further research as a result of the NIF publication. Adjusting for applicants and application differences, awardees had greater scientific productivity than non-awardees. Recipient of the NIF award was three times (OR = 3.4; 95 % CI = 1.9, 5.9) more likely to publish the NIF-submitted application than non-recipients. Award recipients also had greater success in establishing further research with secured funding than non-recipients. These findings suggest that through an effective and targeted approach of the seed funding programs, the novice investigators might succeed in better scientific productivity and production of further research with new initiatives. The effect of small granting foundations in promoting junior researchers’ scientific career is also shown by other investigators [5, 7–9] but in more specific targeted disciplines. Bawden et al. [5] found that the Canadian Association of Emergency Physicians small grant program was important in study completion and scholarship promotion of the award recipients. The National Institutes of Health (NIH) small grant program for behavioral research in cancer control [9] found tailored small grant programs to be beneficial for promoting and retaining new investigators’ scientific productivity.

The positive effects of funding awards on the scientific productivity of novice investigators suggests that (1) the peer-review committee was effective at accurately allocating the limited research funds; (2) small sums of seed money were important in completing and publishing the initial research of novice investigators, securing funding for further research, gaining professional recognition and collaboration, and advancing the new faculty’s scientific career. The personal comments of the awardees on the effect of the NIF award in their scientific career attests to the effective operation of the NIF program. For example, five large randomized controlled trials with secured funding were initiated by award recipients compared to one large randomized controlled trial by a non-award recipient.

We found that gender, affiliation and discipline, or expertise of the applicants did affect the funding decision, but the level of education was an important factor in allocating funding. This finding was expected as although these factors were not considered in decision-making; research education was one of the aspects of the ranking process. Physicians with graduate education and applicants with a doctorate degree were funded more than other applicants. Although it might seem that pediatrics received less funding than cardiology and surgery, it was study design and methodology that affected the funding decisions. More than 50 % of cardiology and surgery proposals were RCTs or pilot/feasibility trials, while more than 50 % of the pediatric proposals were observational studies. The reason might be that (1) it is hard to randomize pediatric populations due the feasibility and ethical issues such as population vulnerability and gaining guardian approval and (2) not all comparative research questions are feasible or appropriate for random allocation [10]. To increase the chance of funding, researchers should engage rigorous methodology to control bias when randomization is not feasible. The level of education also had an effect on the publication of the research in the NIF applications. Overall, applicants with Master’s and Bachelor’s degrees published less often than other applicants (p = 0.038, Table 2). This might suggest that these applicants might require a more committed mentor to provide guidance and ensure completion and publication of their project.

One unique strong aspect of this paper is the qualitative survey data related to why award recipients and non-recipients failed to publish the research in NIF application. The open-ended questions provided detailed information on the applicants’ perceptions, experiences, successes, and challenges of the application progress. Many award recipients indicated that the completion and presentation of the NIF application assisted them in establishing the direction of their research plan, generating further research, and gaining internal and external recognition and collaboration. This does not mean that the NIF award was the core of the succession. These applicants were likely highly motivated and knowledgeable investigators and the provided seed funding triggered their research plan. It should be noted that failure at the NIF level does not necessarily and directly lead to a less successful future academic career. Our evaluation of the non-awardees qualitative survey data revealed interesting findings. Many non-recipients who completed the open-ended questions claimed satisfaction by securing funding from other sources or completing the NIF-submitted research without funding. This might further suggest the importance of small grant funding programs, as the success of many non-recipients was dependent on the scholarship from other small grant agencies. Many non-recipients were neither discouraged nor unsuccessful in advancing their scientific productivity, generating further research, and gaining professional recognition and collaboration. The reasons for not awarding these likely outlying applicants might have been due to ineligibility, program priority, or limited funding. Despite these outliers, many non-recipients had significantly lower scientific productivity than award recipients suggesting the likelihood of effective and productive performance of the NIF process. Some non-recipients were completely discouraged and lost interest in their application. Some found writing grant applications to be challenging and stopped trying. Likewise, five award recipients were also discouraged and discontinued their project due to feasibility issues such as slow recruitment.

Mavis and Katz [7] found similar findings among new investigators of diverse expertise who received seed funding awards to advance their research career. They found that award recipients were productive in terms of the quantity and quality of the publications as well as federal grant support than non-recipients. They also received similar written comments about the significance of the award in their research career. They did not however collect qualitative survey data on the non-award recipients. This current report is unique in collecting and reporting qualitative survey data on both award recipients and non-recipients. The reason for this is that NIF is an internal program and thus we had greater access to applicants for gathering survey data than Mavis and his colleagues [7].

Other strengths of this evaluation are the rigorous methodology and a bias-controlled measurement of the primary and secondary outcome measures. We used a systematic review process and searched, independently and in duplicate, Medline and Pubmed databases to minimize bias in identifying the publication of NIF article, journal ranking, and citation of the article. We also used independent and duplicate processes in categorizing the qualitative data into different themes. The NIF peer-review committee applied independent and duplicate ranking of the applications for funding decision-makings. The reviewers had to justify their ranking by providing the strengths and weaknesses of the applications. We also made extra efforts to maximize the response rate to the electronic survey by sending multiple reminders. We received a high response rate of 90.5 % to the survey completion request. The factors that might have played an important role in this high response rate were as follows: (1) The annual submission of survey on the progress of the application for awardees was a required criteria. (2) We knew many of the applicants and their research mentors. (3) We often collaborated with our trainees and faculty when they leave HHS or McMaster University. (4) Mentors often were senior staff with tenure positions and provided the information when we failed to locate the applicant.

The current report has certain limitations inherent to evaluation studies of this nature. First, this report is retrospective in design with a prospective process, which makes it prone to missing data. Second, the qualitative survey data is self-reported and prone to recall and report biases, which is the case with all surveys. The lack of reliability and validity testing of the survey questions also limits the interpretations of the open-ended survey. There were 28 non-responders despite multiple reminders. Third, we have described the applicants’ discipline to explore its effect on the award decision but failed to adjust the publication of the NIF applications for academic discipline due to the following reasons: (i) Some specialties, whether physician or non-physician, had very small data and pooling them together as one category made the interpretation less meaningful. (ii) It is known that the number of journals and opportunities for publication varies between disciplines, which makes publishing for some specialties more difficult [7]. (iii) The research topic of non-physician applicants was not often relevant to the applicant’s discipline. The NIF applications were mainstream biomedical sciences and the effect might be minimal. Besides, discipline was not considered in the funding decisions. (iv) This is a single-center data with limited sample size. It would be a valuable contribution to the literature if two or more institutions planned these programs with similar guidelines and criteria. (v) We only evaluated the scientific productivity of the applicants but did not quantify their career development and academic promotion because receiving small seed funding may not justify academic promotion. Career development and academic promotion are equally related to educational and administrative activities. (vi) The number of completed and published NIF applications is likely underestimated. The data was collected at one time point in December 2014, and earlier applicants have had more time to complete their projects and publish than more recent applicants. This likely occurred in equal probability for recipients and non-recipients. As indicated by the follow-up survey, some applicants were in the process of conducting or completing the NIF application project. Some had submitted their manuscript or it was in press. To optimize accuracy and avoid bias, we did not report these manuscripts as published.

CONCLUSIONS

These findings suggest the effectiveness of research investment as small as seed funding for scientific productivity and professional growth of novice investigators and production of research with new initiatives and innovations. Further efforts are recommended to enhance the support of small grant funding programs.