Introduction

Many public policy and public scholars hope to affect policy and management practice through their work (see, e.g., Cherney et al., 2012a), and several decades of research utilization research have shown that many policymakers and public administrators do sometimes use research products in their practice (see, e.g., Head et al., 2014; Newman et al., 2016). Indeed, in recent decades, some nations have implemented research assessment systems on the explicit expectation that research contribute to policymaking (Carson & Given, 2021). There are many ways besides direct practitioner uptake of published research products by which academics can affect practice (Lawrence, 2023; Williams & Lewis, 2021). Examples include consulting, informal advice, teaching, general-audience media appearances or contributions, and serving as a practitioner oneself (see, e.g., Landry et al., 2010; Olmos-Peñuela et al., 2014a, 2014b; Spaapen & van Drooge, 2011). Though empirical case studies have illustrated some such mechanisms, integrated understanding of the mechanisms by which academic social science researchers affect policymaking and public administration remains weak (Carson & Given, 2021; Nelson et al. 2023).

A substantial body of prior literature has examined the related question of the types of “productive interactions” involved in effecting societal impact from research or knowledge transfer from researchers (see, e.g., Landry et al., 2010; Molas-Gallart & Tang, 2011; Muhonen et al., 2020; Olmos-Peñuela et al., 2014b; Spaapen & van Drooge, 2011). This approach, rather than seeking to measure specific impacts from research (difficult in the physical and engineering sciences, and even more so in the social sciences) focuses instead on interactions that could plausibly lead to societal impact. As Fotheringham et al. (2021) observe, any individual research project is likely to have only a modest, and perhaps individually undetectable, impact on final policy outcomes. Meanwhile, tracing back specific policy outcomes to the often diffuse and variegated research activities that may have influenced them is a difficult and time-intensive business. Focus on productive interactions seems a promising approach to measure and evaluate policy impacts of public administration and policy research. However, much of this research (e.g., de Jong et al., 2014; Landry et al., 2010) has not focused upon social science research.

Social science has distinguishing characteristics from physical sciences and engineering—e.g., low rates of patent and spin-off production—that may shift the set of knowledge transfer mechanisms at play. Research on social impact mechanisms in the social sciences has (1) generally focused on knowledge transfer or societal impact, which may take many forms besides policy influence (e.g., affecting the operations of firms or facilitating the work of advocacy groups) and (2) not focused on public administration and policy research, which, as research explicitly about public administration, policy, and policymaking, may be expected to be peculiarly relevant to policymaking and public policy practice. Thus, it is an open question whether previously studied varieties of productive interactions correspond to mechanisms by which public administration and policy researchers achieve policy impact.

As an empirical question, we do not know which public administration and policy researchers attempt, and which of those succeed at, policy influence through different mechanisms, nor do we know what factors affect such choices, successes, and failures. These gaps in knowledge make it difficult to, for example, detect disparities in what categories of researchers are able to make themselves heard by what methods, discuss systematically what competencies researchers require in order to successfully influence practice, or posit methods by which research organizations might better induce or support their employees to pursue policy impact (Carson & Given 2021).

This paper addresses these deficits by drawing on survey data from a sample of 409 authors of articles published in major public administration and policy journals since 2015. Based on factor analysis of self-ratings of lifelong policy impact through 16 different mechanisms, we identify four broad categories of activity by which academics achieve policy impact: research uptake, wherein practitioners make use of an academic’s published research; teaching, wherein researchers influence practitioners through education or training; media, wherein a researcher influences practice through general-audience media appearances or coverage; and advice, wherein an academic provides direct advice (paid or unpaid) to practitioners. We refer to these four broad categories of mechanisms as impact channels.

We build on this categorization to offer four regression models exploring determinants of researcher practice impact through each channel. Interestingly, available demographic characteristics have little effect on variation of impact through each channel. Squaring with previous research on determinants of magnitude of research impact, respondents’ career-long motivation to have their research used in practice is the most broadly relevant regressor. This work builds a foundation for further investigation of determinants of researchers’ perceived policy impact by channel and illustrates that such exploration is sorely needed.

This paper follows a slightly unconventional structure due to the two-phase nature of our analysis. The following section, “Background: mechanisms of academic social science impact on practice” discusses prior literature and theory on the mechanisms by which academics affect public policy and public administration and potential associations between such. “Methods” provides information on our survey methods and data.  “Impact channel derivation” discusses the derivation of our four impact channels through factor analysis.  “Background and theory: determinants of academics’ magnitude of impact on practice through each channel” offers a theoretical framework aiming to explain variation in magnitude of impact through the four derived channels, along with derived hypotheses.  “Impact determinant data and results” provides information about respondents relevant to this theoretical framework and presents statistical models intended to test the.  “Discussion” discusses results, limitations, and theoretical implications, and  “Conclusion” concludes and outlines directions for further inquiry.

Background: mechanisms of academic social science impact on practice

As discussed above, though utilization of academic social science research in public administration and policy practice has been studied for several decades, there is little direct precedent for our investigation. Nelson et al. (2023) systematically review the last two decades of English-language literature on the topic and find few quantitative studies investigating social science researchers’ influence on public administration and policy practice through multiple different impact mechanisms. Many qualitative and quantitative studies have, however, illustrated particular mechanisms individually. Furthermore, a substantial body of research has investigated the broader question of how academic researchers generally, and how social scientists specifically, may achieve societal impact. The “productive interactions” literature, drawn from the field of research evaluation, focuses on assessing academic activities that could contribute to societal impact rather than attributing demonstrable impacts to particular research or researchers (which is often difficult and tenuous; see Spaapen & van Drooge, 2011; e.g., Molas-Gallart & Tang, 2011). Thus, it develops a large catalogue of potentially impactful actions. These investigations may, with appropriate caveats—for research also illustrates that academic impact incidence and mechanisms vary across disciplines (Cherney et al., 2013; Ouimet et al., 2010)—inform expectations about the narrower question treated in this paper.

Prior studies have not done much to examine commonalities across mechanisms of impact: for example, whether some different mechanisms draw on similar skills or resources or are used by similar practitioners. Muhonen et al. (2020; see also Morton, 2015) develop a typology of social “impact pathways” for research projects, and others, e.g. Esko and Miettinen (2019), have attempted more narrowly scoped typologies of social impacts for specific disciplines. But though it is conceptually plausible, it remains empirically untested whether different mechanisms tend to group together in any way. In our view, different impact mechanisms—e.g. teaching internships or workshops and connecting with formal students, or offering contributions to traditional or social media—may draw upon similar skillsets and perhaps even emerge from similar dispositions, situations, or sets of responsibilities. Researchers achieving impact through one activity may be expected to possess skills, dispositions, and positions tending to produce impacts through similar activities. Prior investigations on this topic vary in the granularity of the activities treated—from “direct, indirect, and financial” (Spaapen & van Drooge, 2011) to “radio, television” and “school conference” (Jensen et al., 2008). The more granular frameworks have not searched for associations between their granular elements, while the coarser ones have taken such associations for granted. Some frameworks have categorized mechanisms more by conceptual than by empirical association. For example, the common typology of “direct” (i.e. personal, spoken), “indirect” (i.e. written or artifactual), and “financial” (e.g. contracting or contribution in kind) interactions between researchers and societal stakeholders, introduced by Spaapen and van Drooge (2011), does not (and is not intended to) say, for example, whether participating in one form of “direct” interaction means that researchers are more likely to participate in other forms. Similarly, this typology offers no implications as to whether some researchers tend toward some categories of interaction and away from others. There is an open opportunity to empirically test conceptual associations between specific research impact mechanisms.

Only some research impact mechanisms identified in previous literature are likely to be performed by public administration and policy researchers, or to influence policy or public administration practice. For example, public administration and policy researchers rarely produce patents or spin-off firms (both discussed by Landry et al. 2010), while university “open days” (discussed by Jensen et al., 2008) will not influence policymaking save by unusual happenstance. Plausible mechanisms by which public administration and policy researchers could influence public policy or public administration practice include, but are not limited to, researcher partnership or collaboration with policymakers (e.g. Broström & McKelvey, 2018; Cherney et al., 2012a, 2012b), with industry (e.g. Tilbury et al. 2021), or with civil society organizations (e.g. Meagher et al., 2008; Weiss-Gal et al., 2017); researcher teaching and training of practitioners (e.g. Charles, 2021, Tilbury et al. 2021); researcher employment as practitioners (e.g. Weiss-Gal et al., 2017); researcher media appearances or contributions (e.g. Andrews, 2017; Meagher et al., 2008); legislative testimony (e.g. Weiss-Gal et al., 2017); practitioner-initiated use of research (e.g. Cherney et al., 2015); and researcher participation or reference in research curation and evidence review processes (e.g. Bozeman et al., 2019, Castillo et al. 2021; Youtie et al., 2016).

There are several reasons why use of some impact mechanisms might associate with one another. Some academic roles or activities might be conducive to achieving impact in multiple different ways. For example, teaching-heavy positions may be conducive to achieving impact through teaching workshops or internships, as well as through relationships with former students. Research-heavy positions, conversely, may be more conducive to achieving impact through uptake of research products. Moreover, some impact mechanisms may draw upon similar personal skills or inclinations. For example, dissemination of research through conventional media coverage or contributions, or through social media, may draw upon publicity and presentation competencies. All these mechanisms of association might operate independently or in parallel. Empirically distinguishing them will be difficult because they produce similar expectations: that mechanisms tending to flow from activities involving similar work, e.g. different forms of expert advising and consultation, of teaching, of media outreach, and of research product uptake, will tend to associate. We refer to “activities involving similar work” as “functionally similar.”

Other associations are, however, imaginable. For example, different academics might have different overall “styles” of communication with practitioners, by analogy to Shehata et al.’s (2017) styles of academic communication. These authors define three broad styles of academic communication: “orthodox,” characterized by reliance on traditional, formal, heavily vetted channels; “heterodox,” featuring heavy use of informal and contemporary methods; and “moderate,” falling somewhere in between. These categories cannot be cleanly mapped onto policy impact mechanisms, not least because traditional channels of scholarly communication have never been primary channels of communication with public administration and policy practitioners. It may be possible to articulate some parallel set of “orthodox” approaches to policy impact—perhaps including government testimony, producing policy reports through government-sanctioned advisory organizations like the U.S. National Academies of Sciences, Engineering, and Medicine, service on expert advisory committees, contract research, personal career mobility, and, for an elite few, personal relations with policymakers. These might be contrasted with later (though still venerable) approaches of mass-media communications and advocacy group collaboration or service, and finally with web and social media approaches.

As previous literature on “productive interactions” (e.g. Molas-Gallart & Tang, 2011; Muhonen et al., 2020; Spaapen & van Drooge, 2011) tends to group impact activities functionally, we formalize this tendency as a hypothesis for testing.

H1

Mechanisms through which particular researchers achieve impact will group functionally, i.e., mechanisms involving similar work will associate with one another.

Methods

Data and sample design

We draw here upon data from a novel survey of academic researchers in the fields of public administration and public policy, the Truth to Power (T2P) survey. Between November 2020 and February 2021, the survey collected data on research utilization and outreach activities from 409 authors of articles (at that time) recently published in seven major policy and public administration journals. We could not include all high-impact journals in these fields due to resource constraints, so we instead aimed for a purposive selection of prominent journals from across public policy and public administration. Our selected journals display a U.S. emphasis, but this emphasis is by no means exclusive. As shown under “Impact determinant data and results” below, 55.5% of our respondents reported affiliation with a U.S. organization, with most of the remainder in Europe and a few respondents in each of Asia, Oceania, and South America. Any selection of journals for sampling will involve judgment and partiality, and future studies using other journals would be desirable.

The sampling frame included first through third authors of all non-editorial, non-opinion, non-book-review articles published between 2015 and 2019 in five public administration and public policy journals: Public Administration Review (PAR), the Journal of Public Administration and Theory (JPART), the Journal of Policy Analysis and Management (JPAM), Research Policy (RP), and the Milbank Quarterly (MQ). We broadened the sample by adding the most recent 50 articles, again excluding editorials, opinion pieces, and book reviews, published between 2015 and 2019 from each of the American Journal of Evaluation (AJoE), and Evaluation Review (ER). PAR and JPART are high-quality public management journals, JPAM is a high-quality general policy and program evaluation journal, RP and MQ are high-quality journals in science and health policy, respectively, and AJoE and ER are high-quality evaluation journals. This frame produced a total of 826 articles and 1536 authors, including redundancies. Table 1 below displays the initial sampling frame.

Table 1 Initial sampling frame by journal. Adapted from Nelson and Lindsay (2023)
Full size table

We note that the respective journals differ to some degree on the extent to which the full set of articles could be viewed as closely related to public administration and policy. In particular, Research Policy and Milbank Quarterly, as journals focused on, respectively, science and technology policy and health policy, include articles focused not only on public administration and policy but also on management, economics and related fields. However, inclusion of domain-specific policy journals permits investigation of a broader spread of public administration and policy research. All domain-specific journals we examined included a mix of papers focused on public administration and policy and other topics. In follow-up studies, we hope to examine practitioner use of papers directly focused on public administration and policy compared to papers focused on other topics, including not only economics and management but also studies in other social sciences.

Survey instrument and procedures

First authors for each article received email invitations to participate in our survey on Qualtrics at the beginning of the survey period—January 28, 2021. If first authors did not respond within a week, they received a second invitation email and second and third authors received initial invitations as well. After another week, we sent a final invitation to all nonresponding second and third authors. One hundred and eighty-four authors did not receive the invitations due to out-of-date or incorrect email addresses, reducing our sampling frame to 817 articles and 1352 authors. Table 2 below summarizes article response rates by author type. Table 3 summarizes article response rates by journal.

Table 2 Response rates for articles by author type. Adapted from Nelson and Lindsay (2023)
Full size table
Table 3 Response rates for articles by journal. Adapted from Nelson and Lindsay (2023)
Full size table

The survey instrument consisted in three parts. The first part queried authors about use by practitioners of the research presented in the specific article from which their names were drawn. We do not draw on these data in this paper, but these results are reported in Bozeman et al. (2023a, 2023b). The second part queried authors about their career-long experiences and activities with research utilization. The third part collected basic demographic information along with a few additional personal details (e.g., whether respondents had ever worked as practitioners themselves). For each responding author, we also collected publicly available publication and citation metrics.

Impact channel derivation

Impact activity data

Our survey asked public administration and policy scholars to self-rate their career-long impact on public policy or public administration practice through sixteen different mechanisms of impact (Table 4). This list does not include all previously identified mechanisms of social impact for research. Some, as discussed above, do not typically emerge from public administration and policy research—these include patenting, creation of spin-off firms, and creation of “hard” technological products (Muhonen et al., 2020; see also Olmos-Peñuela et al., 2014b). Others do not directly affect policy; these include university public outreach events. Meanwhile, contract research and formal research collaboration can be quite important in attracting use by public administrators and policymakers. However, these processes are often so different from use of researcher-initiated work that we do not include contract research with its “built in” demand.

Table 4 Respondent ratings of their own impacts on public policy or public administration through different activities, ordered descending by mean value
Full size table

Despite a plurality of burgeoning methods, including the “productive interactions” approaches discussed above, assessing the societal impact of research remains notoriously difficult. This is particularly true in the social sciences (Pederson et al., 2020). Interestingly, seven of the top eight impact activities in our data, ranked by mean importance rating, typically involve direct interpersonal contact with practitioners (the exception being “use of [the respondent’s] research by practitioners [without] direct contact with [the respondent] or [the respondent’s] research colleagues). Researchers might be biased to perceive themselves as achieving greater impact through activities involving direct contact with practitioners simply because such contact facilitates awareness of impact. Practitioner use of research without direct contact may be underreported, as use often occurs without researcher awareness (Masood et al., 2020; Weiss, 1980). If this form of common source bias is present, it could inflate association between impact mechanisms involving direct contact (see generally George & Pandey, 2017; Richardson et al., 2009).

Factor analysis

To identify any strong associations between subgroups of impact factors, we investigated the possibility of subsuming the mechanisms under a small set of “impact channels.” As we were interested in major activities by which researchers achieve impact, we omitted all activities for which the sum proportion of the two highest importance ratings did not exceed 20%. This eliminated election to political office, temporary job assignment in a government agency, volunteer work, op-eds, and coverage or contribution on television or radio. To those remaining, we applied a principal axis unrotated factor analysis with varimax rotation, retaining dimensions exceeding one eigenvalue. This procedure produced four factors with fairly strong and distinct loadings for most of the impact activities: our “impact channels” (Table 5).

Table 5 Factor loadings for retained impact activities
Full size table

Highly loaded variables for each factor exhibit a face similarity. Factor 1 appears to cover use of research and informal transmission of ideas to practitioners. Factor 2 covers teaching activities, as well as, somewhat problematically, work through national organizations. The reasons for such overlap may require further investigation. Factor 3 covers media engagement, and Factor 4 covers formal consulting and advice. If present, common source bias through preferential reporting of direct-contact mechanisms, discussed above, does not appear to have dominated the analysis. Direct-contact and non-direct-contact mechanisms are in places grouped together (e.g., under Factor 1), while direct-contact mechanisms are split between the factors. Mechanisms appear to group mostly through functional similarity. That is, mechanisms involving similar activities group together—media with media, consulting with consulting, etc. This result supports Hypothesis 1, favoring the functional-similarity account of impact mechanism associations discussed in Sect. “Methods”.

Background and theory: determinants of academics’ magnitude of impact on practice through each channel

Impact-relevant variables

Following derivation of our four impact channels, we turned to additional survey data to investigate determinants of respondents’ factor scores. As discussed above, this exact question has not been treated before, but work on related and more general questions of academics’ social impact may inform expectations. Prior studies have indicated that personal demographic characteristics such as age and gender; professional attributes such as rank, field, scholarly productivity, and network assets; research project characteristics such as funding source; research group characteristics such as size, multidisciplinarity, and presence or absence of “star scientist” leadership; and organizational features such as size, prestige, and focus on societal impact, among others, may affect to what extent and through which mechanisms a researcher achieves social impact (see generally Jensen et al., 2008; Landry et al., 2010; Olmos-Peñuela et al., 2014a, 2014b). We discuss in detail only those potential determinants for which we have data (Fig. 1).

Fig. 1
figure 1

Generic conceptual model of determinants of a researcher’s impact on practice through an impact channel j

Full size image

Prior research has shown features of individual researchers to covary with extent of impact (or of potentially impactful activities). Such features may include demographic variables such as age and gender, psychological ones such as impact motivation, professional characteristics such as discipline, rank, background, productivity, and skills, and political-geographic ones such as nationality (D’Este et al., 2018; Perkmann et al. 2013, 2021). We possess data on researchers’ use motivations, academic productivity, rank, career length (academic age), gender, and race.

Individual characteristics

Researcher use motivation

Studies have shown that social scientists are more likely to achieve policy impact when personally motivated to do so (Bozeman et al., 2023a, 2023b; Weiss-Gal et al., 2017). One plausible reason for this association is tailoring of the topics, nature, and presentation of research to user contexts, which Cherney et al., (2012a, 2013) show to predict research uptake in policy practice. Another reason may be variation in effort dedicated to policy outreach activities. Muhonen et al. (2020) discuss twelve different ways in which social science and humanities researchers may achieve social impact, all of which require deliberate action on an academic’s part beyond simple publication. Note that use motivation should not be understood as a fixed and wholly internal quality. One may be motivated to achieve research impact for either intrinsic reasons of conviction or personal values, or extrinsic reasons including monetary or prestige incentives. For one analogy, D’Este and Perkmann (2011) recognize several varieties of motivation for industry engagement among UK-based physical science and engineering researchers, including pursuit of research support, information seeking, and even personal pecuniary motives.

Thus, we expect use motivation to associate positively with achievement of impact. However, teaching duties, which can result in policy impact, have no necessary correspondence to impact motivation. It is unclear whether greater policy impact motivation might lead to differences in teaching style, sought positions, or other variables that could alter the likelihood of achieving impact through teaching.

H2

Motivation to achieve impact on policy or public administration will associate positively with achievement of impact through research uptake, media, and expert advising, but not through teaching.

Researcher academic productivity

Academically productive researchers may stand a greater chance of achieving policy impact by virtue of greater media visibility, greater academic reputation, and greater likelihood that some of their many works will find their way to practitioners (Olmos-Peñuela et al. 2014b). Commercial activity has been shown to associate positively with academic productivity (Azoulay et al., 2009; Lowe & Gonzalez-Brambila 2007; Zucker & Darby 1996), and Jensen et al. (2008) find that more productive physical scientists and engineers tend to be more active in industrial collaboration. However, the applicability of these findings to social science policy impact is unclear. More directly relevant, Thomas and Ormerod (2017) find productivity to associate positively with policy impact among tourism researchers. It is possible, however, that above a certain level, dedication to impact may reduce productivity (or vice versa). Perkmann et al. (2021), reviewing studies on physical scientists’ industry engagement, find productivity to associate positively with some types of engagement, but negatively with highly applied varieties such as consultancy.

Distinctions in associations between impact mechanisms have some prior empirical support. Olmos-Peñuela et al. (2014b), in a study of social scientists and humanists, find that research groups led by highly productive “star scientists” are more likely to engage in consulting and contract research, though not in joint research, training, or transfer of personnel outside of the academy. Landry et al. (2010), studying physical scientists and engineers, find teaching and publication to be substitutive, i.e., for more of one to associate with less of the other. Previous evidence is thus ambiguous. We tentatively predict productivity to associate positively with achievement of impact through research uptake, media, and advice, but not through teaching.

H3

Academic productivity will associate positively with achievement of impact through research uptake, media engagement, and expert advising, but not through teaching.

Academic rank

Higher-ranking researchers may stand a greater chance of achieving impact for several reasons. First, higher-ranking researchers may have stronger networks of support and connections to practice, which may facilitate impact. Second, rank may proxy for skills or capabilities conducive to impact. Last, early-career researchers may be less inclined to pursue societal engagement or impact due to opportunity costs of such activity in the publication-focused incentive structure of academia (Braxton & Del Favero, 2002; Diamond, 1993).

Prior studies have indeed shown rank to associate positively with societal engagement in the physical and engineering sciences (D’Este & Perkmann, 2011; Jensen et al., 2008; Perkmann et al., 2021), and with policy impact within the social sciences (Thomas & Ormerod, 2017; Weiss-Gal et al., 2017). Differential effects of rank across different impact mechanisms are less clear. Studying physical scientists and engineers, Landry et al. (2010) find being an assistant professor to negatively associate with teaching (and with scholarly publication), but to show no association with patenting, spin-offs, consulting, or informal knowledge transfer. Olmos-Peñuela et al. (2014b) find social science and humanities research groups headed by full professors to engage in more consultancy, training, and engagement through personnel mobility than do other research groups, but find no association with contract or joint research. Thus, evidence of an effect on impact for higher rank is inconsistent, but, where an effect is present, it is positive.

H4

Academic rank will associate positively with achievement of impact through all mechanisms.

Researcher career length

Though rank does associate with career length, the two are not identical, and rank may not entirely swallow the effects of career length. Landry et al. (2010) find network capital to associate positively with several knowledge transfer activities. Low-ranking but long-serving researchers may benefit from a longer history of work and engagement and may have stronger networks than younger colleagues of equivalent or even higher rank. Age (itself closely linked but not identical to career length) does associate positively with social science utilization in at least one study (Thomas & Ormerod, 2017), but studies of academic engagement generally have found ambiguous effects (Perkmann et al., 2021). We treat career length as a control variable for the purposes of analysis.

Researcher gender

Gender bias may be present in academic and public organizations, potentially making it harder for female or gender-nonconforming researchers to achieve nonacademic impact. Thomas and Ormerod (2017) find gender insignificant in their multi-method study of UK academics’ policy impact. However, Walker et al. (2019b), in their survey study of UK academics, find that women report policy experience at equal rate to men but that men report more extensive engagement with research users. Gender may have differential effects across impact channels. Toutkoushian and Bellas (1999) find men to have lower teaching loads than do women. However, Jensen et al. (2008) find the opposite result and furthermore find women to be more active in “dissemination activities” than men. Landry et al. (2010) find male gender to associate positively with some but not all knowledge transfer activities. Due to associations between gender, career length, and rank, we use gender as a control variable.

Researcher race

Racial bias on the part of practitioners may make it more difficult for researchers of color to achieve research utilization. Race is rarely addressed in studies of researchers’ social impact, though at least one study shows that Black faculty tend to have heavier teaching loads than white colleagues (Toutkoushian & Bellas, 1999). Due to associations between race, career length, and rank, as well as low numbers of nonwhite respondents, we use race as a control variable.

Organizational characteristics

Organizational characteristics in general

Previous studies have shown that individual researchers’ impacts in several different domains systematically vary with the characteristics of the organizations in which they work (and indeed that research organizations’ impacts vary with similar characteristics). These previous studies are not precisely analogous to the question treated here, but they may, again, inform expectations. Literature reviews from D’Este et al. (2018) and Perkmann et al. (2021, 2013) summarize many potentially significant variables. These include organizational support for, normalization of, and rewards for extra-academic engagement, departmental or research group interdisciplinarity, practical support for societal engagement by academics, cultural barriers, and university or department quality. Furthermore, Kitagawa and colleagues (2018) observer that, despite shared systemic pressures toward impact in higher education (see, e.g., Crow & Dabars 2020, Mowery & Sampat 2006, Nowotny et al., 2003), universities have developed a diversity of societal impact profiles (Jacob et al., 2003; Sánchez-Barrioluengo, 2014) shaped, they argue, by heterogeneous “institutional logics.” This is an organization-level argument, but organizational impact profiles are of course composed of individual activities. Other studies corroborate the suggestion that organizational features can shape not only whether but how academics pursue research outside the university (Abreu & Grinevich, 2013, Halilem et al., 2017, Huyghe & Knockaert 2015). Although all these constructs are likely relevant, we have data only on organizational prestige and organizational outreach incentives.

It should be noted that our impact measures are lifelong, and academics often move between universities (and sometimes other organizations) over their careers. Thus, inasmuch as university characteristics affect researcher impacts, individual researchers may experience different such effects over their careers. It could, in principle, be possible to query academics about the length of their stay at each university in their history and construct an index of average lifelong university quality to match with lifelong impact; or to query academics granularly about their impacts while at each university in their personal histories. However, either of these approaches would substantially complicate data collection and analysis while likely offering little meaningful gain. As is, we simply use data on affiliations listed on the publications from which we constructed our dataset. These organizations will for some be longstanding homes; our respondents tend senior. Moreover, respondents’ current organizations may in general proxy for the characteristics of academics’ organizational homes over time, given high-prestige universities’ tendency to hire from other high-prestige universities (Wapman et al., 2022).

University quality

Previous evidence on effects of university quality on individual researchers’ scholarly engagement or impact is ambiguous. D’Este and Patel (2007) find little effect of departmental quality on academic-industry engagement, save in applied disciplines, where the relationship is negative. They suggest two possible reasons. They first turn to methodological weaknesses in the UK Research Assessment Exercise, from which they draw quality ratings. Second, they suggest that researchers at lower-quality universities may be more willing to focus on firms’ immediate problems. Ponomariov (2008) finds a negative association between organizational academic quality and propensity of physical scientists and engineers to interact with the private sector—the reverse of the pattern on the organizational level, where industry interaction and academic quality associate positively. Ponomariov proposes that researchers working in more prestigious organizations may be more incentivized to focus on basic, peer-reviewed research than on applied work.

In contrast, Landry et al. (2010) find that size of both organization and research unit—often used as proxies for quality—associate positively with commercial impacts and publication. These authors posit that greater size reflects greater access to resources for knowledge transfer. However, they find organizational and research unit size to associate negatively with teaching activities, and they do not theorize this relationship. Scheulke-Leech (2013) finds positive effects of departmental quality on industry engagement, suggesting that access to high-quality human resources may facilitate engagement. Libaers (2014) finds no effect for university research intensity rank on engagement, Johnson et al. (2017) find no effect of university research intensity on academics’ commercialization intentions. In review, Perkmann et al. (2021) suggest that weaker universities may experience more resource scarcity, incentivizing researchers to look to extra-academic partnerships for funding. All these articles have dealt solely with the physical sciences and engineering. In the social sciences, Olmos-Peñuela et al. (2014b) find research group size to associate positively with consultancy and contract research, but not with joint research, training, or personnel mobility. They argue that greater resources help to attract nonacademic interest.

Noting that the negative associations have been found only for commercial activities by physical science and engineering personnel, whereas positive associations have been found in the social sciences, we suggest that social science researchers at higher-quality universities are more likely to achieve policy impact through most activity categories thanks to greater visibility in the policy practice sphere and to greater access to resources for knowledge mobilization. However, we observe that, while resources for and incidence of publication, media interaction, and consultancy all may vary with resources and prestige, all universities teach; and faculty at lower-quality universities often have heavier teaching duties. Faculty at higher-prestige universities may be more likely to teach eventual high-ranking government officials, but our impact measure does not distinguish between impacts at different levels or ranks of government. Thus, we do not expect an association between university quality and impact through teaching.

H5

University quality will associate positively with impact through research uptake, media engagement, and consultancy, but not with impact through teaching.

Incentives for impact

Prior work has more straightforwardly suggested that an organization’s cultural support (Bercovitz & Feldman, 2008; Clarysse et al., 2011; Stuart & Ding, 2006) and rewards for extra-academic engagement (Kenney & Goe, 2004; Tartari et al., 2014) associate positively with such engagement. Previously surveyed and interviewed social scientists also cite lack of support or incentives as a serious barrier to pursuing policy impact (Cherney et al., 2012a; Matthews et al., 2018). We include organizational use incentives in our conceptual model but not in our statistical one, as we expect any effect to be mediated through individual use motivation. Prior analysis, available upon request, has shown use incentives to have no detectable effect on impact independent of use motivation.

Impact determinant data and results

Descriptive results

Our survey collected basic demographic and professional information about our respondents (Table 6). Respondents tended male, white, and toward higher academic ranks, and the overwhelming majority held PhDs. Approximately ¼ had previously been full-time public policy or public administration practitioners for at least 12 months.

Table 6 Respondent demographic information
Full size table

As indicator for researchers’ academic productivity, we use the Hirsch index, or h-index, a commonly used measure in scientometrics (see, e.g., Egghe, 2010, Jensen et al., 2008, Mingers 2009). H-index—defined as the value h for which h of a scholar’s papers have at least h citations each, and the others have no more than h citations each–is robust against extremes of per-paper citation in a scholar’s record. This is useful in some ways—it does not punish researchers for producing papers with few citations, provided they also produce higher-cited ones—but it also is not sensitive to a small number of papers with extremely high citation rates. H-index makes no adjustment for career length; Qiu et al. (2008) observe that the measure is not well-suited to measure young researchers’ impacts. This is not a major problem because our respondents tend older, with a median value of 12 and a mean value of 15.4 years since completing their highest degree. Moreover, we include career length as a control variable. Average h-index values (like citation rates themselves) vary systematically across fields, rendering inter-field comparison difficult (Malesios & Psarakis, 2014). However, owing to our journal-based sampling methodology, most of our respondents reside in the closely related fields of public policy or public administration. This scope reduces the significance of inter-field comparison problems. Thus, while h-index, like all metrics, has particular weaknesses, it provides a good measure for our purpose, that is, within-field comparison of senior scholars’ career-long publication visibility.

We use university rankings as indicators of university quality—specifically, the 2021 Shanghai Ranking list. Although departmental characteristics may more directly indicate the resources of academics’ immediate organizational environments (see, e.g., D’Este & Patel, 2007), use of department-level data introduces difficulties in (1) identifying which of several departments public administration and public policy researchers may belong to (to say nothing of joint appointments) and (2) potential difficulties in cross-comparison between rankings of different department types. There is precedent for influence of university-level indicators on research impact as well (Landry et al., 2010). Meanwhile, while university rankings are very imperfect (Vernon et al., 2018), they are still the only available multinational, quantitative indices of university quality. Shanghai Ranking’s Academic Ranking of World Universities is a well-established and transparent ranking. Note that lower rank number indicates higher quality, so association signs will be reversed.

Our survey also queried respondents about their overall motivations to achieve research use and their home organizations’ incentivization of use. Respondents tended to place significant priority on achieving research use, though they reported only modest rewards for such from their home academic units (Table 7). It is, of course, possible that use motivation self-report might be subject to social desirability bias. This bias might operate universally across impact mechanisms; there is no clear reason that it might operate distinctly between them.

Table 7 Respondents’ ratings of their motivations to achieve research use by practitioners over the course of their careers; and of present organizational incentives to achieve use of research by practitioners
Full size table

Regression models

We present four simple ordinary least squares (OLS) regression models with robust standard errors, based upon the theoretical framework outlined above, to test potential determinants of impact magnitude by channel (Tables 8, 9, 10, 11).

Table 8 OLS regression results for Factor 1: Research Uptake
Full size table
Table 9 OLS regression results for Factor 2: Teaching
Full size table
Table 10 OLS regression results for Factor 3: Media Engagement
Full size table
Table 11 OLS regression results for Factor 4: Expert Consultation
Full size table

Regression models partially support Hypotheses 2 and 3, but not Hypothesis 4. Hypothesis 5 is fully supported. As expected, impact motivation associates positively with impact through research uptake and through consultation, and not through teaching; against expectations, however, it does not associate with impact through media (H2). As expected, academic productivity associates positively with impact through media and advising, and not through teaching; against expectations, it does not associate with research uptake (H3). Academic rank does not associate with any impact mechanisms except for teaching, where the association is, as expected, positive (H4). As expected, university quality associates positively with impact through research uptake, media engagement, and consultancy, but not with impact through teaching (H5).

Discussion

Our results display certain important limitations. The items that our factors condense ask respondents to rate the career-long impact on public administration and policy that they have achieved through some particular mechanism. We do not know to what extent the response to such an item reflects actual impact achieved, subjective perception of the importance of that mechanism, and effort expended in the pursuit of that mechanism. A low rating may indicate that the respondent has not attempted a given mechanism, or has attempted it and been frustrated. A high rating may indicate that the respondent has expended a great deal of effort on that mechanism over time with regular and modest impacts; has put in a small amount of effort and gotten a few major returns; or even just that the respondent has put a great deal of effort into that mechanism independent of actual impact.

Fortunately, this limitation does not significantly hinder interpretation of mechanism grouping. Our results strongly suggest that public administration and policy researchers affect public administration and policymaking through (at least) four broad and roughly separable functional categories of activity: research uptake, teaching, media engagement, and expert consultation. That is, impact mechanisms involving similar work tend to associate with one another. Empirical clustering of more granular mechanisms into these four categories helps to validate purely conceptual grouping of these activities in previous analysis (e.g., Landry et al., 2010; Olmos-Peñuela et al., 2014b). Further investigation will be required to determine why similar impact activities associate. It is not yet known whether different mechanisms cluster due to role associations, skill or personal inclination associations, generational consonances, or other reasons—though the clustering of social and traditional media, for example, may militate against the generational hypothesis.

Our regression models, while achieving only limited predictive power, do corroborate some relationships identified in more general studies of societal impact of research generally and of social science in particular (Table 12). As expected, university quality associates positively with impact through all channels save teaching, which aligns with previous work illustrating the importance of organizational prestige (e.g. e.g. Landry et al., 2010; Olmos-Peñuela et al., 2014b). As expected, motivation to achieve impact associates positively with impact through research uptake and through expert advising, but, contrary to expectations, not through media. Meanwhile, academic productivity associates positively with achievement of impact through media and through expert advising, as expected, but, unexpectedly, not with impact through research uptake. This is an interesting finding, as on its face this suggests that greater academic visibility of output does not entail greater visibility to practitioners. This is plausible—few practitioners read academic journals (Amara et al., 2004; Head et al., 2014; Ouimet et al., 2010)—but it is then unclear why productivity would positively associate with achievement of uptake through media or through consulting. Last, academic rank associates positively only with impact through teaching. Of our control variables, both career length and white race associate negatively with impact through media, while male gender associates negatively with impact through research uptake.

Table 12 Implications of regression models for hypotheses about determinants of impact through different channels
Full size table

We have no certain explanation for these findings on demographic associations, though they might be connected with the perceptual nature of our impact measures. Respondents were asked to rate how “important” each impact activity had been over the course of their career. It is possible that older academics might be less likely to rate media impacts as important because they have had other impacts which they regard as more important, even if their absolute level of impact through media exceeds that of younger colleagues. Similarly, it is possible that white academics might experience greater opportunities than nonwhite colleagues in nonmedia or nonresearch venues, thus rate media impacts as less important. These are only speculations, however; it is equally plausible that these results derive from omitted variables or other limitations of our data.

Improvement upon our models will likely require explanatory recourse to features of academics, of their home organizations, and even of the political cultures and government organizations with which they interact not included in our dataset. Prior research has illustrated that the nature and incidence of research utilization varies across policy domains (Landry et al., 2003; Newman et al., 2016), academic disciplines (Desmarais & Hird, 2014), national contexts (O’Brien, 2005; Pattyn et al., 2019), the skills of the researcher (Walker et al., 2019b; Weiss-Gal et al., 2017), and the extent of organizational support for academic outreach to practitioners (Lightowler & Knight 2013; Weiss-Gal et al., 2017). We expect that these and perhaps other factors will need to be integrated into any robust explanation of variation in researchers’ impact on practice by mechanism.

There is, however, reason to be cautiously optimistic about the generalizability of our results across the subfields of public policy and public administration, and across national contexts. Our sample could not encompass all aspects of public policy and public administration, but it did draw from a diverse spread of prominent journals across general policy, science and technology policy, health policy, public administration, and evaluation. Our sample displays a U.S. bias, with 55.5% of respondents affiliated with U.S. organizations. The United Kingdom is second-most-represented at 6.4% of the sample. Geographical disparities are more easily summarized by continent. 57.7% of respondents affiliate with North American organizations and 33.7% with European organizations, giving both North America and Europe substantial representation in our sample—though there is, of course, substantial heterogeneity across the nations in each of these regions. Generalizability to nations in Asia, Oceania, South America, and Africa, with, respectively, 5.8%, 2%, 0.7%, and no representation in our sample, must be still more tentative. Relevant variations in national context may include political culture, the nature of relationships between universities and national governments, and the existence and strength of incentives for researchers to pursue policy impact, e.g., national research evaluation programs such as the Excellence in Research for Australia framework and the UK Research Excellence Framework.

Nevertheless, many reasons for functional-similarity grouping of impact channels, e.g., skill commonalities, may be expected to hold across academic fields and national contexts. Moreover, our elucidated impact channels are broad enough to subsume a fair degree of inter-field or international variability (e.g., in the nature of consulting, advising, or teaching arrangements). Similarly, the theory behind our supported hypotheses about determinants—that impact motivation, academic productivity, and university quality engender impact—also seems likely to transfer across fields and contexts. As discussed above, our findings are consistent with the broader literature on societal impact in other academic fields. Limits on generalizability are more likely to come in potential additional impact channels found mostly outside our geographic or disciplinary focus, in variation in the relative importance of different impact channels, in variation in the implications of researchers’ personal demographics for their impact prospects, or in regionally relevant impact determinants not included in our current model (e.g., perhaps, religion, political affiliation, or caste). Accordingly, we expect that our results are likely to remain somewhat valid across scholarly fields and geographic regions. However, future empirical tests of this intuition will be required.

Conclusion

Clearly there is a great deal yet to be learned about how public administration and policy academics affect practice and why different academics do so through different mechanisms. We have validated four broad categories of mechanism through which academics influence practice—research uptake, teaching, media engagement, and expert consultation. These categories appear to be functional ones. That is, impact mechanisms involving similar work tend to cluster together. However, the reasons that functionally similar impact activities tend to associate with one another will require further investigation. Greater knowledge of what constitutes functional unity will also help to explain the presence of “hybrid” mechanisms split across our derived factors—e.g., the split of “responding to former students’ requests for ideas” across “research uptake” and “teaching.” Future studies could examine the degree to which our impact channels can travel well across disciplines, and perhaps add in additional impact mechanisms for analysis.

Nonetheless, the analysis is useful as it stands. It puts empirical weight behind functional groupings of activities commonplace in previous studies of scholars’ research impacts both within and without the social sciences. It also extends these grouping approaches to the specific case of impacts on policy by public administration and policy researchers. Social science impact in general receives less attention than does impact of physical science and engineering research (for a review of exceptions, see Reale et al., 2018). Further investigation could do well to integrate different approaches to impact with different varieties of impact—e.g., the commonly used taxonomy of conceptual, instrumental, and symbolic use of research in policymaking (Amara et al., 2004; Weiss, 1979, 1991). Different types of interactions may be more appropriate to different types of use. For example, teaching might be more apt to produce “conceptual” use—general shaping of the terms in which policy problems and options are understood—and consulting to produce “instrumental” use, i.e. application of research to specific decisions. It would also be valuable to investigate different mechanisms of impact at the organizational rather than the individual level (e.g., Barbara et al., 2021).

We have also developed regression models investigating factors affecting magnitude of impact achieved through each mechanism. Our most robust findings are that researcher use motivation and greater organizational prestige are conducive to achievement of impact through most impact channels. These findings also align with prior work on more general societal impact of the social and even of the physical sciences, indicating that, to some extent, policy impact and public administration research can be fruitfully addressed using theoretical frameworks developed for these more general problems. Additional empirical and conceptual work, however, will be required to investigate variation in relevant factors across different impact mechanisms. Theories describing determinants of general impact exist (D’Este et al., 2018; Perkmann et al., 2013), and empirical work has previously treated multiple different impact mechanisms (e.g. Jensen et al., 2008, Landry et al., 2010, Olmos-Peñuela et al. 2014b). But more mechanism-specific theories have not been articulated. In particular, it will be necessary to investigate not only how characteristics of individual researchers and research organizations affect varieties of policy impact, but how characteristics of government organizations also affect modes of communication between the two sectors (Althaus et al., 2021). It may also be useful to study how different forms of organizational support for research-policy engagement affect what impact methods academics use, and with what degree of success (see generally Hopkins et al., 2021).

Thus, we know that public administration and public policy researchers achieve impact through several broad, functional channels, but not in much detail which researchers use which channels. Varieties and magnitudes of impact achieved may vary across different impact mechanisms, and different academic attributes or situations may be conducive to achievement of impact through different mechanisms. Further inquiry and more robust understanding could be useful not only for impact researchers but for academics and managers aiming to identify and improve impact.