Introduction

Completing successful scientific research that offers a unique, significant contribution to a research area often depends on the researcher’s ability to characterize the literature to identify a topic that furthers the development of the area and is of interest to the academic community. Many methods exist to identify, analyze, and synthesize the literature, which range from focusing on the content of the literature, to focusing on the way in which the research is being conducted and the characteristics of the publications themselves (Hood and Wilson 2001; Cronin et al. 2007; Higgins and Green 2011; Patra et al. 2006; Taylor and Taylor 2009; Smith 2012). For example, the field of scientometrics aims to analyze the published literature using quantitative or statistical techniques to evaluate the impact of an area while research synthesis approaches are focused on evaluating the content of publications to obtain meta-inferences across studies (Hood and Wilson 2001; Walsh and Downe 2005; Cooper et al. 2009; Cooper and Hedges 2009). While these approaches provide valuable insights that guide research, the level of development of the research area may have an impact on the trustworthiness of the results from these types of analyses.

The notion of the state of the research area is often referred to as its level of maturity—essentially, whether the research area is new and highly theoretical or has been more developed with some convergence on best practices (Cheon et al. 1993; Maloni et al. 2009; Neely 2005). The concept of the maturity of a research area is commonly included in literature analysis but it has not yet been formalized and a brief review of the literature on research area maturity reveals that, while many authors investigate some form of this concept, the criteria and assessment approaches used vary widely. Recent trends in research synthesis techniques have led to more systematic, repeatable approaches to literature reviews, including the broader use of analysis methods such as bibliometric analysis (Hood and Wilson 2001; Patra et al. 2006; Schoepflin and Glänzel 2001). In order to assess maturity in this type of review, the concept must first be clearly defined, as there is no currently agreed-upon definition, and operationalized in a way that allows it to be compatible with existing approaches and easily applied as part of a standard analysis procedure. In addition, the procedure must be flexible enough to be applied in any research area, which would allow for a consistent approach and may support effective comparisons between different research areas. Due to the inconsistencies in the application of this concept, a comprehensive review is needed to identify the full breadth of criteria used to assess research area maturity.

This paper presents the results of a study that uses a systematic literature review (SLR) to identify the literature, bibliometric analysis to assess the current state of this area, and research synthesis techniques to identify the criteria used to assess research area maturity. The results are used to develop a framework to formalize the concept so that it may be more systematically evaluated and easily incorporated into existing approaches for analyzing the literature. In the following sections, the literature on research field maturity is discussed followed by a description of the bibliometric results and the proposed framework. A case example is also briefly discussed in which the proposed framework is applied to assess the maturity of the Engineering Management field (Keathley et al. 2015, 2016). Finally, implications of this work and areas for future research are discussed.

Background

One of the most critical aspects of conducting useful research is correctly framing the study, which generally consists of identifying the literature area, analyzing the publications, and synthesizing the research content (Cooper and Hedges 2009; Cooper et al. 2009; Cronin et al. 2007; Patra et al. 2006). This process provides the foundation for new research and allows for the positioning and defense of the work. Generally, some form of a literature review is used to identify the specific publications that represent a research area. Once the literature is identified, it is often analyzed to identify trends or characteristics (Hood and Wilson 2001; Patra et al. 2006; Schoepflin and Glänzel 2001). For example, the field of scientometrics focuses on analyzing the published literature to evaluate the impact of the research area such as bibliometric analysis, which uses quantitative or statistical assessments to describe the publications (Pritchard 1969; Brookes 1990; Hood and Wilson 2001; Schoepflin and Glänzel 2001; Patra et al. 2006; Garfield 2009; Smith 2012). Finally, some form of research synthesis is often applied, which focuses on integrating or interpreting the content of the research itself as opposed to the characteristics of the publications to create meta-inferences that are more generalizable (Borenstein et al. 2009; Cooper and Hedges 2009; Walsh and Downe 2005).

Recent trends in research framing have resulted in the emergence of much more rigorous methods including approaches that focus on quantitative analyses (Cronin et al. 2007; Hood and Wilson 2001; The Cochrane Collaboration 2008). One example of this is the SLR, which was originally used in the medical field where studies tend to be very consistent in both terminology and reporting (Cronin et al. 2007; Higgins and Green 2011). This approach has been further developed in recent years, making it appropriate for use in any research area, including areas where publications are not as consistent (“Campbell Resource Center,” n.d.; Tranfield et al. 2003). In addition to using a stronger method to identify the literature, methods in scientometrics have become more commonly used to analyze the literature with bibliometric analyses being conducted in many research areas (Hicks 1999; Hood and Wilson 2001; Patra et al. 2006; Schmenner et al. 2009; Schoepflin and Glänzel 2001; Archambault and Larivière 2010). Through the use of a more rigorous review of the literature and a more complete analysis of the characteristics of the literature, research framing is becoming a much more defined and rigorous process with many researchers combining approaches to obtain a more comprehensive understanding of a research area.

The maturity of a research area, as mentioned previously, is commonly addressed in literature analyses. However, the concept lacks a consistent definition and firm set of assessment criteria. Still, instances of this type of assessment do share some basic characteristics, such as focusing on changes over time and determining the level of maturity through evaluations of the literature (Cheon et al. 1993; Neely 2005). The assessment of maturity generally aims to analyze the current state of the research area, identify current and future trends, and provide more in-depth evaluation of a research area as well as justify areas for future research (Budi et al. 2013; Porter and Detampel 1995). While many of these assessments are based on scientometrics approaches, this approach goes beyond the typical literature analysis to lend further insight into how well established the field is and the relative trustworthiness of the conclusions drawn from the literature. In general, a research area can be described as progressing from a highly conceptual stage where most of the research is exploratory, to a more advanced stage where quantitative studies are conducted, best practices are identified, and prescriptive information is disseminated (Cheon et al. 1993; Maloni et al. 2009; Neely 2005). While all research areas experience some form of this progression, each area develops uniquely and at different rates, further complicating this type of assessment (Cheon et al. 1993). Conversely to the notion that there is a single progression, some authors aim to define research cycles or lifespans such as Neely’s evolutionary cycle (Carlile and Christensen 2004; Neely 2005); it can be argued that the number of cycles (or the phase of the research) indicates the maturity of the field. Another maturity characteristic that is commonly mentioned, but rarely methodically investigated, is the relationship between academic research and typical practice in the field. Some argue that the goal of academic research is to solve real-world problems and generate practical solutions (Gagnon and Ghosh 1991; Maloni et al. 2009, 2012; Pasqualine et al. 2012). Therefore, a research area should evolve from an exploratory beginning to conceptual frameworks being proposed and tested, and then to industry exposure and finally, a convergence on best practices and consistent terminology (Neely 2005; Stone 2012).

The assessment of research area maturity mainly focuses on analyzing the literature and, therefore, best fits within the analysis or measurement phase of research framing. Many of the tools and approaches used to evaluate maturity criteria are based on bibliometric techniques including approaches such as co-citation analysis, impact factors, science mapping, and evaluation of indices for scholarly output or impact (Small 1973, 1999; Garfield 2006; Diallo et al. 2016). However, a review of the literature suggests that some criteria for maturity also imply identifying supplemental literature or information such as data related to the infrastructure built to support the research area, including professional societies, funding, and academic programs or courses (Paul-Hus et al. 2016; Borrego and Bernhard 2011; Moody 2000; Sud and Thelwall 2014; Bornmann 2015). In addition, some criteria involve synthesizing the content, such as analysis of themes or construct definitions during the synthesis phase of the framing process (Cobo et al. 2011; Harvey and Myers 1995; Taylor and Taylor 2009). Therefore, it appears that, although an assessment of the maturity of a research area is primarily focused on analyzing the literature, it is a broader approach that includes aspects of both scientometrics and research synthesis. An investigation of the literature reveals that while maturity is being addressed in some works, the methods used to analyze maturity are not standardized and focus instead on a variety of characteristics including authorship, publication trends, topics, and methods used (Cheon et al. 1993; Maloni et al. 2009; Neely 2005; Becheikh 2010; Grover 2012; Houy et al. 2010; Nissen 1995; Sun et al. 2015; Piro et al. 2016; Zhao and Zhao 2016). In addition, a review of the literature suggests that the emphasis placed on the different dimensions of maturity is unbalanced and some of the more difficult criteria are only lightly discussed, while more concrete ones, such as scholarly output and impact factors, are more widely addressed.

While many studies focus on applying a selected set of criteria to assess maturity, some researchers have attempted to define frameworks for this type of assessment (Cobo et al. 2011; Nie et al. 2009; Wendler 2012). Typically, these approaches identify a prescribed set of criteria and apply bibliometric or complementary tools to enable the assessment. In addition to describing the current or historical state of the research area, some studies also focus on forecasting future trends or identifying areas for future work (Budi et al. 2013; Porter and Detampel 1995). There are also a few examples of researchers developing automated tools to support the assessment process or investigating ways to compare analyses across disciplines (Li et al. 2009; Drew et al. 2016). While many of these studies propose an approach to analyzing the literature, they typically focus on a narrow set of criteria that are relevant to the area being assessed. In order to ensure a sufficient understanding of the research area maturity, a distinct definition of research area maturity in conjunction with an assessment approach that easily works with existing systematic methods is needed. Such an assessment can provide many important insights to support research framing as well as a unique perspective when analyzing the literature. In addition to having a stronger foundation for making inferences from the literature analysis, specific information regarding the types of studies that would create a more mature research area can be utilized to identify more effective objectives for future research (Maloni et al. 2012; Neely 2005). A structured assessment approach based on a comprehensive framework could also provide a more standard comparison across fields to further support the development of an area based on the maturation process of more developed research areas (Maloni et al. 2012; Perianes-Rodriguez and Ruiz-Castillo 2016).

Methodology

The first phase in analyzing the literature is to identify the set of publications that represent the research area of interest. In many bibliometric studies, this is accomplished through some form of systematic review where a platform or database is searched and the full results are then evaluated based on a set of criteria to identify the publications specifically related to the area. Because the study of research area maturity has not yet developed into an independent field, there is no one representative source that can be used to identify these publications. Therefore, a comprehensive SLR was conducted to identify the literature for this analysis. This approach was chosen considering the inconsistencies concerning terminology and application of this concept, and to ensure that the review was rigorous and an accurate representation of the literature across disciplines. The SLR process, shown in Fig. 1, was adapted from both Tranfield et al. (2003) and the approach presented in the Cochrane Handbook (Higgins and Green 2011; Tranfield et al. 2003). To further increase the rigor of this study, the approach was executed with a team of researchers, which introduced more consistency in the development of the search strategy, application of exclusion criteria, analysis of the publications, and synthesis of the assessment criteria (Higgins and Green 2011; Tranfield et al. 2003).

Fig. 1
figure 1

Systematic literature review approach

Full size image

The primary objectives of this research, as described previously, were focused on evaluating the current state of this research area using several bibliometric analyses and identifying the full breadth of criteria investigated in the literature through a content synthesis in order to create a general framework that can be used to assess maturity in any research area. To begin, the scoping study was conducted during which the research team identified eight papers that included explicit criteria for the maturity of a research area (Cheon et al. 1993; Grover 2012; Houy et al. 2010; Maloni et al. 2009; Neely 2005; Nissen 1995; Pasqualine et al. 2012; Taylor and Taylor 2009). Next, databases and platforms (online services that include multiple databases such as the Web of Science) were evaluated to determine the sources to be included in the search. Prominent platforms that cover a wide range of topics and document types were chosen as they had the potential to provide a wider range of results. The final set of platforms consisted of the Web of Science, EBSCOhost, and ProQuest. In each of these platforms, all of the available databases were included in the search to allow for publications from all research areas to be included in the results. However, the document types were restricted to only academic works (i.e., journal publications, conference papers, books, e-books, and dissertations/theses) due to the nature of the concept of research area maturity, which is primarily relevant to the academic community.

The focus of this review was then decomposed into three primary search concepts: research area, maturity, and assessment. The papers in the scoping set were then used to identify terms and phrases that could be used as search terms (the terms entered into the databases) for each concept and a Boolean phrase was constructed and tested including evaluating several search tools, such as truncation and proximity operators. The final strategy consisted of searching all fields instead of a full-text search meaning that the publication would have to have the search terms as a central focus of the paper and included in fields such as the title, subject, keywords or abstract in order to be captured by the search. This served to restrict the search results to only highly relevant publications. It was found that one term, “the field,” should be included as an exact phrase while the other search terms with more than one word were searched using the NEAR operator with a tolerance of four words allowing for more flexibility in the search. For example, the search term “research NEAR area” returns phrases such as “research area” and “area of research” (Table 1).

Table 1 Search terms
Full size table

Once the strategy was defined and the Boolean phrase finalized, the search was executed on the three platforms and initial results were tabulated as shown in Table 2. The results were then limited based on the document type and limited to only English-language publications. In addition, the duplicates resulting from publications being indexed on multiple databases within a platform were removed. It is important to note that ProQuest is able to run the search and provide a number of raw results but also has a limit on the number of results that it can process and, therefore, the search on this platform was broken into three subsets. After limiting the initial results and removing all duplicates, the initial exclusion criterion, i.e., excluding papers that were only loosely relevant to the assessment of maturity, was applied by evaluating the titles and abstracts of all of the limited search results. This resulted in 614 relevant papers, including all papers in the scoping set. Figure 2 shows the number of papers identified from each platform, which clearly shows the unique contribution of each platform and supports the use of multiple sources for a more comprehensive review. This comparison across platforms has also been investigated in other recent studies (Mongeon and Paul-Hus 2016).

Table 2 Search results
Full size table
Fig. 2
figure 2

Initial exclusion criterion results

Full size image

Once the results were reduced to the initial set of 614 publications, two additional exclusion criteria were applied by a detailed review of the publications. First, any papers that mentioned some form of this concept or used similar terminology but did not evaluate or address the concept in any concrete way were removed and, finally, each publication was evaluated to ensure that it included at least one criterion for the assessment of maturity. This criterion either could be explicit or could be implicitly represented either in the interpretations or in a portrayal tool used in the analysis. This process resulted in 123 papers, which were considered the final paper set to be analyzed in this work. The complete list of citations for these papers can be obtained from the authors upon request.

Results of the bibliometric analysis

A bibliometric analysis of the resulting set of 123 papers was conducted to determine some of the basic characteristics of the literature. The results of this analysis provided insight into the extent of academic focus in this area and what research perspectives support the development of this area. Figure 3 presents the publication trends, which shows a general increasing trend over the past several years with the initial papers published in the 1980s. It is important to note that this review was conducted in 2014, which influenced the relatively low number of papers identified in that year. It is clear that the past decade has seen a sharp increase in publications that explicitly focus on maturity assessment.

Fig. 3
figure 3

Publications per year

Full size image

Next, the types of publications were investigated and it was found that 89.4 % of the 123 publications were from academic journals. This is somewhat expected as the concept of maturity assessment is generally included in literature reviews and academic journals are an appropriate outlet for that type of research. In addition, the results show that the remaining 10.6 % of the publications, consisting of conference papers, books, book chapters, and dissertations/theses, all occurred between 2001 and 2014. This result, coupled with the trend in publications per year, suggests that the concept of maturity is increasingly being addressed and supports the assertion that this topic is being studied more broadly in recent years. Figure 4 summarizes the five most frequent publication sources represented in the final paper set. In addition to the number of papers published each year, the journal impact factor at the time of publication is also provided. The results show that Scientometrics Journal is the most common publication source represented, which is expected due to the nature of assessing research area maturity. However, it is interesting to note that the other four top journals are specific to application areas. In fact, this is true for 14 of the top 15 most common journals represented in the final paper set. Finally, it is interesting to note that the journal impact factor at the time of publication varies widely among the papers found in these journals (Table 3).

Fig. 4
figure 4

Publication source topic area

Full size image
Table 3 Impact of publication sources
Full size table

The outlets that the papers were published in were also evaluated and categorized according to the discipline or topic area associated with that outlet, as shown in Fig. 4. The results again show that the publications come from a variety of research areas, clearly demonstrating the interest for the concept from various disciplines and underlining the need for a common framework that supports analysis and knowledge sharing across different research areas.

Next, the authors were investigated to identify any prominent authors or author groups in terms of scholarly output. The results show that there were 308 unique authors with only eleven authors having published more than once on this topic. Authors that published more than one paper in this set are identified in Fig. 5. The eleven authors represent five groups of co-authors coming from various disciplines, as shown in the figure, and 14 academic papers. One thing that they have in common is that each of the 14 papers is some form of literature review that has analysis of the maturity, development, or evolution of the field as a primary focus of the paper. In addition, two of the author groups use the term ‘maturity’ to describe the concept (Grover and Carter/Maloni) while the other authors use alternative terms. Another interesting aspect of these papers is that only one group (Pou-Merina, Mazarron, Canas-Guerrero, and Calleja-Perucho) explicitly identify their work as a bibliometric analysis while the other papers typically include metrics associated with this type of analysis but do not explicitly identify it as a methodology. While many of the research groups applied these concepts to multiple research areas, some focused on assessing a single area over time providing a more robust assessment of the development process (Cheon et al. 1993; Grover 2012; Lim et al. 2007).

Fig. 5
figure 5

Prominent authors

Full size image

In addition to investigating the number of papers per author, the author’s country was also tabulated and the results are shown in Fig. 6. It is important to note that 41.6 % of authors were from the United States, which has been removed from this figure for readability. The authors in this set of publications were found to come from 29 countries with only countries with more than one author shown in this figure. It is interesting to note that, while Europe and North America are predominantly represented, there are authors from all inhabited continents represented. This result, coupled with the increasing trend in publications, suggests that research including a maturity assessment is becoming more common across a variety of research communities.

Fig. 6
figure 6

Author country

Full size image

To further investigate the impact of the publications in the final paper set, the average number of citations per year was investigated. Table 4 summarizes the results of the ten most highly cited publications from this research area. It is important to note that, while some of the publications in the final paper set discussed the maturity of an area as secondary focus, all of the most highly-cited publications are directly focused on evaluating the development of a field. In addition, the results further demonstrate the broad set of disciplines that are focused on conducting this type of assessment.

Table 4 Most highly-cited publications
Full size table

Finally, the research designs were investigated to identify the types of review that were conducted and the areas that were the focus of these reviews, as shown in Fig. 7. The results showed that 19.5 % of the final paper set explicitly used the term ‘maturity’ while the remaining papers assessed the concept of maturity but used other terminology such as development or evolution. Next, the results showed that 50.4 % of the publications identified bibliometric analyses as at least part of their methodology. This supports the previous assertion that, while many maturity studies include this type of analysis, there are significant proportions that do not explicitly apply bibliometric analyses and opt, instead, for a wide range of alternative methods. Next, the publications were evaluated to determine what type of review was conducted to better understand how the concept of maturity has been applied in the literature. The results show that the most common approach used in these papers is the systematic literature review, which has become more common in the past two decades. It is also important to note that 20 % of the studies were not literature reviews and, instead, defined a conceptual framework or used an alternative method to address the concept of research area maturity. Finally, the results suggest that many researchers have begun to focus on defining a specific journal or set of multiple journals as the sources for their reviews in the past decade. The variations in research design characteristics demonstrate the need for an explicit definition of research area maturity and a comprehensive framework that is adaptable to different disciplines and research designs.

Fig. 7
figure 7

Types of reviews represented

Full size image

The research area that the review was focused on was also investigated and categorized as shown in Fig. 8. The results show that, similarly to the journal categorization, the research areas studied varies widely but seems to be focused more on business and technology fields. One research area that is not as prevalent in this assessment as expected is the healthcare field. As mentioned previously, many of the systematic methods that are applied in these papers originated from the healthcare field and it is interesting to see that the concept of research area maturity is not as common in this area. Another interesting result is the focus on assessing the maturity of the field of scientometrics. As a relatively recently emerging research area, assessing the maturity could lend important insights and significant contributions to this area.

Fig. 8
figure 8

Research areas reviewed

Full size image

Maturity assessment framework

Once the general analysis of the final paper set was complete, each paper was evaluated by two researchers and the data concerning the criteria identified in the papers were collected. The criteria used to assess the research area maturity were collected including investigating the metrics and portrayal tools in addition to explicit criteria defined in the papers, ensuring that any implicit criteria were also captured. To ensure consistency during the collection of the criteria data, two researchers were assigned to independently collect the data for each paper. Similarly to previous phases, inconsistencies in the collected data were resolved through group evaluation. The resulting database consisted of approximately 1200 criteria, metrics, and portrayal tools.

In addition to identifying the criteria used, the papers were evaluated to determine if any reported an explicit definition of maturity. The results showed that only one paper (Karuga et al. 2007) reported a definition for maturity and it was centered on the concept of the “extent to which [a research area] has built a cumulative knowledge base.” In fact, it was found that most of the publications either implicitly assumed the definition of maturity was understood, or applied a more general definition of maturity related to the concept of reaching a desired state of development. In general, the results of this study suggest that the maturity of a research area describes how and to what extent the area has developed over time with particular interest in the creation, growth, and dissemination of knowledge. Maturity should be assessed with multiple holistic criteria that evaluate both the existing literature as well as other aspects related to infrastructure and diffusion practices. In general, a mature field is one that:

  • is well-documented (i.e., codified) and broadly accessible;

  • is agreed upon by a distinct research community;

  • is differentiated from other research areas;

  • is robust across research paradigms, research methods/approaches, contingent factors, and application contexts;

  • has an impact on the research community, i.e., is cited by other research areas; and

  • is put into practice (i.e., diffusion to industry).

Using this definition of research area maturity, an inductive synthesis approach was conducted that consisted of several iterations of independent reviews and group discussions. As mentioned previously, the data were collected using the terminology and orientation that the original authors used. Once the raw data were collected and organized, the criteria were coded and organized into criteria and sub-criteria and then further into dimensions, which reflect the primary concept underlying a group of criteria.

The resulting framework is shown in Table 5, which consists of nine dimensions of research area maturity and 24 unique criteria that can be used to assess the level of maturity. This table includes the dimensions of maturity, the related criteria for each dimension, the sub-criteria and related “drill-down” options, and example metrics. It is important to note that many metrics exist for each sub-criterion and the metrics listed in this table are selected examples of commonly-used ones. More obvious metrics, such as “number of” or “proportion of” the criteria, are generally omitted from the table.

Table 5 Maturity assessment framework
Full size table

When applying this framework, researchers first need to determine the type of review that the assessment will be based on and then decide which dimensions and criteria are appropriate for their assessment. In addition, many criteria are concerned with analysis of the identified literature but some criteria, particularly in the diffusion and infrastructure dimensions, are focused on supplemental information that must be collected independently from the results of a literature review. Criteria that are focused on analyzing the papers resulting from the review are indicated with a ‘P’ while criteria that are based on supplemental information from outside of the resulting paper set are indicated with an ‘O.’ Once the applicable portions of the framework are defined, researchers should select the criteria and metrics that are most appropriate for their study. One important aspect of this framework is that each criterion can have a different level of maturity. The maturation of a research area can be highly complex and each indicator should be assessed independently and then the results considered together to obtain an understanding of the overall maturity level.

The results of this study offer a comprehensive set of assessment criteria that classifies existing individual tools and approaches into a structured framework, which can guide researchers in conducting a more complete analysis of a research area. It is important to note that the results of this study suggest that, while the definition of a ‘mature’ research area should be relatively consistent across research areas, the interpretations of criteria and, in some cases, even the applicability of certain criteria may vary. As noted previously, each research area will mature in a unique way and the assessment should be tailored to the area for an accurate assessment through careful consideration of the interpretation of the criteria. For example, one common criterion for maturity is the presence of interdisciplinary research teams and collaboration. This is much more relevant in scientific fields than it would be in a humanities field where the convention is to have single-author papers. In addition, the purpose of this study was to identify a comprehensive set of criteria for maturity and not all of the criteria, or even dimensions, are applicable to every research area. Finally, the subset of the framework that is applicable will also be heavily impacted by the type of review that is being conducted. For example, a literature review of a single journal cannot assess criteria such as the breadth of journals represented in the set, and dimensions such as the genesis of the area are not assessable by time-period limited reviews.

Case example

In order to test the proposed framework, a pilot example was conducted to assess the maturity of the field of Engineering Management (EM) (Keathley et al. 2015, 2016). This field has grown in recent years with a significant amount of published research and journals dedicated to the area. The Engineering Management Journal (EMJ) was selected as the focus of this investigation as it includes both academic and practitioner publications from a wide range of EM topics. Further, in order to understand the most recent advancements in this area, the scope of this analysis was limited to the last 10 years of publications.

To begin, the most recent 10 years of publications in the EMJ were collected, which consisted of 40 issues and 227 publications. Due to the scope and structure of this assessment, the Maturity Assessment Framework was evaluated to determine which dimensions and criteria were applicable in this case. Since the review consisted of only one journal and a limited time-period, the Genesis of the Area dimension and the Publication Outlets criterion were not applicable. In addition to the 227 publications identified for the assessment, additional information was obtained to assess criteria related to the Diffusion and Infrastructure dimensions such as academic programs, journals dedicated to the area, and communities of practice related to EM. The analysis of the Authorship Characteristics, Publication Characteristics, Research Design Characteristics, and Impact dimensions were previously reported (Keathley et al. 2015, 2016).

The results of applying the framework suggest that the maturity of the EM research area varies based on the criterion being assessed. Due to the scope of this paper, only selected results are discussed. First, the analysis of Authorship Characteristics focused on evaluating the Author Quantity, Author Productivity and Collaborations among the authors. There were 451 unique authors identified with only 83 having published more than one paper in this area. The results showed that the authorship in this area is relatively concentrated with a small set of core authors that publish most often, with a regular influx of new authors per year. This suggests that the area is relatively well developed with a stable set of ‘experts’ that publish in this area. The country of origin, institutions and disciplines of the authors were also investigated to demonstrate the collaboration in this area. The results showed that the authors represented 33 countries, 153 institutions, and 57 unique disciplines. Interestingly, approximately 18 % of the authors identified their discipline as EM, suggesting that the field has matured to a point where it is a stand-alone discipline. The results also showed that there is a significant amount of collaboration among disciplines including between academic researchers and practitioners. This result supports many claims that EM is a multi-disciplinary field with strong ties to practice (Keathley et al. 2015, 2016). The impact of the publications and authors was investigated by calculating the average number of citations per year, which resulted in the identification of the top eight most-cited publications and the six most impactful authors. The Research Design Characteristics were also investigated and the various data collection and data analysis methods were evaluated. The results showed that there is a wide range of analyses being used but more fundamental methods, such as conceptual frameworks, literature reviews, and case studies, are used much more often than action research and more advanced methods such as statistical hypothesis testing. In addition, there is evidence of an increase in the rigor of the research in terms of longitudinal and multi-sample studies. These results suggest that the methodological aspects of EM research can be improved and advancements should be made by incorporating more rigorous methods and the inclusion of more advanced mixed-methods studies. Finally, by looking outside of the paper set, the team was able to identify academic programs and courses, communities of practice, social media groups, dedicated journals, bodies of knowledge, and certification programs that are dedicated to the area of EM. This suggests that the field has matured beyond simply being an area of academic research and has begun to diffuse into industry both in terms of generating new Engineering Managers through academic programs and in terms of developing EM knowledge and skills that can be used in practice.

The results of this case example demonstrate the applicability of this approach to a research area that is currently developing and in a moderate stage of maturity. The example emphasizes the flexibility of the framework including selecting the portions of the framework that are most applicable based on the research area and the structure of the review used to identify the literature. By using the Maturity Assessment Framework, a more comprehensive understanding of the development and level of maturity of this field was obtained than could have been accomplished by using a SLR, bibliometric analyses or research synthesis alone.

Conclusions

The results of this study support the initial interpretation that the literature concerned with research area maturity is dispersed and assessments of this concept vary. Although several maturity assessment studies were identified, there is little consistency among the studies in terms of assessment criteria and terminology. The review was able to identify a broad set of publications, which resulted in a framework that is much more comprehensive than existing approaches as well as more flexible in terms of being applicable to any research area. In addition, a more relevant and explicit definition of maturity was developed through evaluations of the final paper set. This work contributes a comprehensive review of the concept of research area maturity as well as an operationally-defined approach for assessment. As discussed, the inclusion of this approach in the research framing process combines previously-segregated approaches to analyzing the development of a research area and provides a more directed approach to literature analysis as well as insights into the quality of the knowledge available in the research area and ways to advance the area.

Future work

There are many areas of future work that can help to further operationalize the framework and develop a more structured approach to evaluating maturity. First, the framework should be further refined and tested in a variety of research areas to provide a more comprehensive and robust assessment, including expanding and improving the assessment methods and metrics. Development of an explicit method to rate each criteria as well as a detailed rating system (i.e., developing a rubric for interpreting criteria instead of just assigning a rating of low, medium, or high maturity) are also needed. In addition, a methodology to aggregate the ratings into an overall maturity score would provide a more concise approach to reporting and interpreting the findings. This should also include a method to weight each criterion instead of having them represented as having an equal impact on the overall maturity rating. Future research could include an expert panel or Delphi study to determine the weights to be used in the framework. This type of study could also be repeated for various research areas, which would provide more detailed information on the differences in interpretations and applications among research areas. It would also further support the comparison of maturity across research areas allowing researchers to determine strategic approaches for developing the area. Finally, a concise portrayal tool is needed in order to represent the result from each portion of the assessment. Since each criterion can have a different level of maturity, a radar chart could be a useful option that would allow each criterion to be rated as having a low, medium, or high level of maturity.