1 Introduction

Although the subject of quality did not originally attract much attention in the mixed methods literature (Bryman 2006b; Sale and Brazil 2004), considerable interest has developed in recent years, resulting in the publication of a number of works suggesting criteria for critically appraising mixed methods inquiry in journal articles (Creswell and Tashakkori 2007; Wisdom et al. 2012), research proposals (Creswell et al. 2011), and research in general (O’Cathain 2010). The imperative to address the issue of quality in mixed methods research has been emphasised in several overviews of the field (Creswell 2015b; Teddlie and Tashakkori 2010) and is also evidenced by the inclusion of chapters devoted to this topic in recent key textbooks (Creswell 2015a; Curry and Nunez-Smith 2015).

Addressing quality in mixed methods research is important for several reasons. First, researchers need criteria to ensure that their own studies as well as the manuscripts they assess are warranted and transparent (Collins et al. 2012). This point is especially relevant in the emerging field of mixed methods, as certain fundamental principles are still unfamiliar to many scholars (Tashakkori and Teddlie 2010). Second, criteria can be very helpful for novice researchers both for learning purposes and to avoid the shortcomings more experienced researchers are aware of. And third, the literature on the quality of mixed methods contributes to consolidating the narrative about the distinctive features of this type of research (Creswell 2015b). Indeed, since mixed methods research involves unique procedures—such as the integration of quantitative and qualitative components—there is a case for the use of specific criteria to evaluate the quality of this type of inquiry. Identifying criteria to assess quality and formalising them in frameworks or tools is an undeniable sign of the maturity of the field.

Despite its increasingly acknowledged importance, the issue of quality in mixed methods research faces a number of challenges. Moreover, addressing threats to quality in mixed methods studies can be more difficult than in monomethod studies, due to the greater complexity of the former. As Collins et al. (2007) note, the problems of representation and legitimation associated with qualitative and quantitative research are exacerbated in mixed methods research “because both the qualitative and quantitative components of studies bring to the study their own unique challenges” (p. 268). The operationalisation of quality—i.e. the determination of criteria—can also be difficult in mixed methods inquiry. According to Bryman (2006a), for instance, the specification of criteria may be hampered by the fact that outcomes derived from mixed methods studies are often unpredictable. The same author contends that a researcher might anticipate a set of criteria adequate to his or her mixed methods design, but may well end up requiring other criteria due to unexpected developments in how the study unfolds.

Another challenge for the establishment of criteria has to do with the plurality of views within the mixed methods community, and more particularly, regarding how quality should be conceptualised and appraised. What is perceived as research quality is highly context-dependent: two scholars from different intellectual communities or disciplines may have a different understanding of what quality is or how it should be assessed (Collins et al. 2012). While this diversity of perspectives is a healthy sign of the open-mindedness of the field, it may hinder reaching a minimum agreement regarding definitions, terminology, and procedures for establishing quality in mixed methods research (O’Cathain 2010). As Bryman (2014) points out, the intention to be comprehensive and to reconcile different perspectives in the mixed methods community has generated lengthy lists of quality considerations that are unlikely to be widely used because they are too cumbersome to apply in practice.

Given these challenges, it is important to critically review the literature on the quality of mixed methods research so that opportunities can be identified regarding an agenda for the future. Bearing in mind that discussions about quality are still in their infancy (Leech and Dellinger 2013), this is an opportune moment to put forward recommendations for moving the field forward.

To date, three systematic reviews of the literature have been conducted on the topic of the quality of mixed methods research. The earliest review of the literature (up to 2001), authored by Sale and Brazil (2004) and published in Quality & Quantity, aimed to identify “criteria to critically appraise the quality of mixed-method studies in the health literature” (p. 351). Since the authors did not locate any specific criteria, they suggested a list of quantitative and qualitative criteria for appraising each of the corresponding components separately. Subsequently, O’Cathain et al. (2008), following a literature search conducted in 2006, proposed a set of quality questions and applied them to a sample of mixed methods studies in the field of health services research. These authors found that several of the criteria suggested—such as the assessment and description of the integration of the quantitative and qualitative components—were specific to mixed methods research. More recently, Heyvaert et al. (2013) identified quality frameworks for mixed methods research published up to 2009 and provided an overview of the corresponding criteria. These authors observed that most of the frameworks were published between 2004 and 2009, indicating that the issue of quality in mixed methods inquiry is a “very contemporary one” (p. 313).

Bearing in mind that more than 6 years have passed since 2009—the end of the literature search period in Heyvaert et al.’s (2013) study—it would seem appropriate to update knowledge on the important topic of the quality of mixed methods research. This review aims to do this by answering the following three questions: (1) What literature is available on the quality of mixed methods research and what characteristics are reflected? (2) What criteria for assessing the quality of mixed methods research are suggested? (3) What recommendations address current gaps and points of contention in the literature on the quality of mixed methods research?

Apart from updating the works of Sale and Brazil (2004), O’Cathain et al. (2008) and Heyvaert et al. (2013), this article makes three additional contributions: (1) it provides an overview of the features and trends of the literature on the quality of mixed methods research; (2) it addresses quality dimensions other than the identification of criteria, namely, the terminology used to refer to quality concepts, and the desirability or otherwise of a consensus regarding quality criteria; and (3) it proposes a number of recommendations for a future agenda on this subject.

The remainder of this paper is structured as follows. The methods used to conduct the review are described in Sect. 2. An outline of the characteristics of the literature on the quality of mixed methods research published to date is presented in Sect. 3.1. A summary of the most prevalent quality criteria suggested to date is provided in Sect. 3.2. Current gaps and points of contention regarding quality in the mixed methods field are discussed and recommendations to address them are proposed in Sect. 3.3. Finally, Sect. 4 contains our conclusion.

2 Methods

2.1 Search strategies

A total of 14 electronic databases reflecting social, behavioural and social science disciplines were searched for peer-reviewed journal articles (including editorials and commentaries), books, and book chapters in English. The databases, searched from inception, were as follows: ASSIA (1987–2016), CINAHL (1981–2016), EMBASE (1980–2016), ERIC (1966–2016), International Bibliography of the Social Sciences (1951–2016), MEDLINE (1946–2016), PAIS International (1972–2016), PILOTS (1871–2016), PsycINFO (1806–2016), Science Citation Index Expanded (1945–2016), Social Sciences Citation Index (1956–2016), Social Services Abstracts (1979–2016), Sociological Abstracts (1952–2016), and Worldwide Political Science Abstracts (1975–2016). Boolean search terms associated with the concepts of mixed methods and research quality were used. Terms related to the concept of mixed methods were those suggested by Morse and Niehaus (2009), as these have been successfully used in previous studies on the prevalence of mixed methods research in specific disciplines (Ivankova and Kawamura 2010; Plano-Clark et al. 2008). Although certain terms like ‘multimethod’ and ‘multiple methods’ have explicitly been differentiated in the literature from the term ‘mixed methods’, Morse and Niehaus (2009) contend that, since some authors continue to use the former as synonyms, it is important to use these terms when searching the mixed methods literature in electronic databases. As for search terms associated with the concept of research quality, these were derived from a preliminary review of selected publications on the quality of mixed methods research, which were scanned for terms used to refer to quality. Table 1 lists the strings used for our search, initially performed in November 2014 and updated in February 2016.

Table 1 Search strings
Full size table

The database search was complemented by five parallel strategies. First, a hand search of 14 methodology journals known to have published works on mixed methods research was performed in BMC Medical Research Methodology, Educational Researcher, Field Methods, Forum: Qualitative Social Research, International Journal of Multiple Research Approaches, International Journal of Qualitative Methods, International Journal of Research and Method in Education, International Journal of Social Research Methodology, Journal of Mixed Methods Research, Qualitative Health Research, Qualitative Inquiry, Quality & Quantity, Qualitative Report, and Qualitative Research. Titles and abstracts of articles, editorials and commentaries published in those journals were reviewed, yielding two additional articles that were in press and so not indexed at the time of the electronic search. Second, reference lists of key publications on the quality of mixed methods research were checked. Third, a citation search of those publications was performed in Google Scholar and Web of Knowledge. Fourth, the publication lists of key mixed method authors were downloaded from their academic webpage and reviewed. When no academic webpage was available, the list of publications was sought and downloaded from Google Scholar. Key authors were identified on the basis of having been early developers of mixed methods research as identified by Leech (2010), members of the Editorial Board of the Journal of Mixed Methods Research, or contributors to any of the two editions of the SAGE Handbook of Mixed Methods in Social & Behavioral Research. Finally, a search for grey literature and dissertations was undertaken in CORDIS Library, ProQuest Dissertations and Theses Global, Social Science Research Network eLibrary, and System for Information on Grey Literature in Europe. These additional searches were initially implemented in December 2014 and were updated in February 2016.

2.2 Selection process

From the database search 10,539 publications were generated, from which 6511 duplicates were removed, leaving a sample of 4028 publications for screening by title and abstract according to two inclusion criteria: (1) the publication had to be a theoretical source or an empirical study on the topic of the quality of mixed methods research or had to clearly suggest quality criteria; and (2) the publication had to conform to definitions of mixed methods research as per Creswell and Tashakkori (2007) or Johnson et al. (2007). From this screening, a further 3898 publications were discarded, reducing the sample to 130 publications, whose eligibility for the review was determined after accessing and reading the full text. Applying the same inclusion criteria stated above, 101 publications were excluded on the basis that they were not about quality, or referred only briefly to this topic, leaving 29 publications for inclusion in the review. The same inclusion criteria were also applied to the publications identified by the five parallel search strategies, and these yielded 35 additional records. A total of 64 publications were thus included in this review. A flowchart of search strategies and the selection process is presented in Fig. 1.

Fig. 1
figure 1

Publication search and selection process

Full size image

2.3 Data extraction and synthesis

Different strategies were used to answer the review questions. The first review question was answered by collating the following information: year of publication, discipline of the first author, country of affiliation of the first author at the time of publication, type of publication, nature of the publication, the main purpose in relation to quality, the concepts used to refer to quality, whether quality criteria were reported or not, and how these criteria were arrived at.

To answer the second review question, we used the three-step qualitative metasummary technique as proposed by Sandelowski and Barroso (2007). First, mentions of quality criteria in each publication were extracted and edited into complete sentences. Next, sentences describing an identical criterion were grouped together so as to produce a shorter list of criteria, each of which was checked against the initial publication to ensure that meaning was accurately conveyed. Finally, the frequency effect size—a measure of the relative magnitude of each criterion—was calculated as the number of publications referring to a given criterion divided by the total number of publications suggesting criteria.

The third review question was answered using the framework display technique as proposed by Ritchie et al. (2003). An initial thematic index of issues referring to the quality of mixed methods research was first developed after reading key publications. Publications were then uploaded to NVivo 10 data analysis software for coding using the thematic index, further modified during coding. NVivo’s framework matrix was used to cross-tabulate each publication with the coding index components. Finally, coded content for each pair of items was critically examined in order to gain a general overview of issues both discussed and ignored in the literature.

3 Findings and discussion

3.1 Review question 1: what literature is available on the quality of mixed methods research and what characteristics are reflected?

The 64 publications included in this review are profiled in Table 2. Findings concerning the year of publication reveal that very few works were published before 2005 and, of these, only the article by Caracelli and Riggin (1994) appeared in the 1990s. Despite this being the first paper to address the issue of the quality of mixed methods research, by the end of the 1990s it had only been cited four times according to Google Scholar. This, one might argue, is indicative of the little importance attached to this topic at that time. It also contrasts with the attention dedicated, during the same time period, to other mixed methods topics, such as the characteristics and types of mixed methods designs and the rationale for conducting a mixed methods study—as summarised by Leech and Onwuegbuzie (2009) and Creswell and Plano-Clark (2011). From 2005, however, we see a marked increase in the number of publications on the quality of mixed methods research, with 23 works published from 2005 to 2010, and 38 from 2011 to 2016. One might infer this rise to reflect the exponential growth of mixed methods methodological discussions since 2006, as reported by Ivankova and Kawamura (2010). It might also be related to the influence on subsequent publications of the quality frameworks suggested by Teddlie and Tashakkori (2003) and Sale and Brazil (2004).

Table 2 Characteristics of publications by time period
Full size table

As can be seen in Table 2, across the three periods (pre-2005, 2005–2010 and 2011–2016), most of the publications were journal articles, books and book chapters. Nonetheless, since 2005, the number of other publication types—such as dissertations, technical reports, and journal editorials—have increased slightly, most especially from 2011 to 2016. With regards to discipline, most publications represented the education and health and medicine sectors. Works originating in anthropology, information systems, and management—predominantly published from 2011 to 2016—were less represented. As for the country of affiliation, publications from the USA were the most prevalent across the three periods, followed at a distance by those from the UK, Canada, Australia, Belgium, Finland, Norway, and New Zealand, in that order. Publications from these latter countries, with the exception of the UK and New Zealand, were mostly published from 2011 to 2016. These findings for both the discipline and the country of affiliation are consistent with the review by Ivankova and Kawamura (2010), who found that most of the mixed methods methodological discussions published between 2000 and 2009 pertained to education and health, and were primarily written by authors working in US universities.

Lastly, the table also illustrates the nature and main purpose of each publication in relation to quality. Publications were classified as empirical when they used systematic methods of data collection and analysis, or as theoretical when they relied primarily on the authors’ opinions or other authors’ discussions on the quality of mixed methods research. Overall, the findings demonstrate that, despite the relatively high number of empirical studies published in the first period, since 2005 the vast majority of publications have been theoretical. Although the primary purpose in both empirical and theoretical publications was to develop quality criteria, this was approached in different ways. While the empirical publications generated criteria by means of consultations with researchers (Bryman et al. 2008), analyses of journal reviewers’ feedback (Onwuegbuzie and Poth 2016), or systematic reviews of the literature (Sale and Brazil 2004), the theoretical publications elaborated criteria primarily on the basis of the authors’ personal reflections. Consequently, most quality criteria suggested in the literature are theoretically based rather than empirically derived, reflecting a tendency in the mixed methods literature to construct typologies in theoretical terms rather than derive them from practice (Bryman 2006a).

A final but no less important observation regarding the most recent period has to do with the fact that a significant number of publications pursued objectives other than to develop quality criteria, namely, to assess quality for a given disciplinary field, to reflect on quality, or to test criteria. This would indicate that, overall, the topic of the quality of mixed methods research is moving from a stage of procedural development to one of advocacy and expansion (Creswell and Plano-Clark 2011).

3.2 Review question 2: what criteria for assessing the quality of mixed methods research are suggested?

Table 3 shows the 19 quality criteria with an effect size ≥15 % identified from 35 of the 64 publications included in this review. The remaining 29 publications were excluded for any of the following reasons: (1) they did not suggest criteria; (2) the criteria they suggested was below the effect size ≥15 %, or (3) or they presented a duplicate framework of criteria published elsewhere by the same authors. An example of duplicate framework is Teddlie and Tashakkori (2009) who published an earlier version of the same framework in Tashakkori and Teddlie (2008). Table 3 is organised according to four phases (planning, undertaking, interpreting, and disseminating/applying findings), corresponding to the study phases described by O’Cathain (2010).

Table 3 Quality criteria with frequency effect size ≥15 %
Full size table

The findings—corroborating Heyvaert et al’s (2013) review—show that the type of criteria suggested in the literature are not restricted to procedural aspects specific to mixed methods research, but also include aspects concerning the quantitative and qualitative parts of the study as well as generic research criteria. In fact, the most frequently cited of the 19 criteria is the criterion regarding the quality of the quantitative and qualitative strands—in all four phases, but most specifically in the undertaking phase. Also noteworthy regarding the undertaking phase are two other criteria focusing on the quality of the individual components. These are, first, linkage between the procedures implemented in each strand and the study aims and research questions (ranked fourth for the undertaking phase and sixth for the phases overall), and second, a detailed description of those procedures (ranked fifth for the undertaking phase and seventh for the phases overall).

With regards to generic research criteria, several of these are among the most prevalent criteria in three of the four study phases: specifically, the clarity of the philosophical assumptions of the researcher (ranked second for the planning phase and fifth for the phases overall), linkage between mixed methods design and study aims and research questions (ranked fifth for the undertaking phase and seventh for the phases overall), reporting transparency (ranked first for the disseminating/applying findings phase and ninth for the phases overall), and consistency between inferences and study findings (ranked first for the interpreting phase and tenth for the phases overall). In the overall ranking, the positions of the criteria focusing on the quality of the individual components and the generic criteria are consistent with observations made by a number of authors (Bryman et al. 2008; Greene 2007; Ivankova 2014) to the effect that the quality of the individual parts of the study, as well as of those common to any research inquiry, are a necessary condition for implementing a high quality mixed methods research project.

As for criteria specific to mixed methods research, several of these rank highly in Table 3. This is the case for the criterion on effective integration of the quantitative and qualitative components of the study (ranked second for the undertaking phase and second for the phases overall), and on the provision of a rationale for using a mixed methods design (ranked first for the planning phase and third for the phases overall). In their review of the literature up to 2009, Heyvaert et al. (2013) similarly found these two criteria to be the most frequently cited of specific mixed methods research criteria. Their findings, and those presented in this review, are consistent with the argument that the entirety of the mixed methods literature—and not just the literature concerned with the quality of mixed methods research—points to both these criteria as key principles of mixed methods practice. This echoes Molina-Azorin’s (2011) observation to the effect that integration is a foundational principle and, hence, the cornerstone for the added value of mixed methods research. It also reinforces Bryman’s (2007) statement that mixed methods designs are not universally applicable; rather, when a mixed methods design is used, a clear rationale must be provided.

Finally, other criteria specific to mixed methods research refer to the description of the unique insights gained from using a mixed methods design (ranked second for the disseminating/applying findings phase and tenth for the phases overall), a statement of inconsistencies arising between the inferences emerging from each component (ranked second for the interpreting phase and twelfth for the phases overall), and incorporation of inferences derived from quantitative and qualitative findings in metainferences for the whole study (ranked third for the interpreting phase and 13th for the phases overall).

The findings discussed above lead us to reflect on two additional points regarding Table 3. First, despite the similarities indicated above, important differences exist between publications regarding several elements, specifically, the approach to quality, the terms used to refer to quality and quality criteria, and timing regarding when quality should be assessed. Regarding timing, while some authors have suggested appraising quality in all steps of the research process (Onwuegbuzie et al. 2011; Teddlie and Tashakkori 2009), others have proposed frameworks for assessing quality for completed studies (O’Cathain et al. 2008; Pluye et al. 2009). Second, there are evident similarities between our list of criteria and Bryman’s (2014) core criteria for assessing the quality of mixed methods research. Bryman contends that most quality frameworks existing to date are too large and too complex to use and are therefore not useful for inexperienced researchers. To palliate this state of affairs, he proposes a parsimonious and easy-to-use list of six core criteria to assess quality. Interestingly, all six criteria feature in our Table 3. This convergence of criteria strengthens Bryman’s proposal and also shows that, despite differences in authors’ perspectives on quality, a set of shared criteria can be identified across disciplines and geographical contexts.

3.3 Review question 3: what recommendations address current gaps and points of contention in the literature on the quality of mixed methods research?

Findings in Table 2 show that, since 2005, the topic of the quality of mixed methods research has received a great deal of attention. The steady increase in the number of publications appearing each year suggests that, within the mixed methods community, the issue of quality is viewed as both important and necessary. This review, nonetheless, demonstrates the existence of several gaps and points of contention that need to be addressed for the field to be able to move forward. Three major recommendations that outline an agenda for future discussion and research regarding the quality of mixed methods research are described immediately below.

3.3.1 First recommendation: increase the number of empirical publications on quality

As Table 2 indicates, most publications on the quality of mixed methods research to date have been theoretical, whereas empirical works in the form of discussions with researchers have been few; notable among these, however, have been the contributions of Caracelli and Riggin (1994) and Bryman et al. (2008). This paucity of empirical studies on researcher feedback is problematic since, according to Hammersley (2007), “the best place to start thinking about the nature and role of assessment criteria is with how researchers actually go about judging quality in doing their work” (p. 298). It is therefore likely that the current literature on quality may not accurately reflect researchers’ existing practices, thinking and decisions. We strongly encourage future studies that take into account researchers’ views on how quality should be conceptualised and operationalised, as this would not only improve the usefulness and applicability of existing quality frameworks but would also lead to the inclusion of criteria and other aspects not taken into account to date.

3.3.2 Second recommendation: achieve greater consistency in quality terminology

In the literature, a critical point of contention relates to the terminology that should be used when discussing the quality of mixed methods research. A number of authors have suggested that the language of quality should be based on new terms that specifically reflect the distinctive nature of mixed methods research, contending that these terms should be different from those used in quantitative and qualitative research and yet broad enough to be inclusive of both approaches. This approach has been endorsed by Teddlie and Tashakkori (2003) and Onwuegbuzie and Johnson (2006), who proposed, respectively, the terms ‘inference quality’ and ‘legitimation’ as substitutes for the term ‘validity’. The fact that the language of quality in social research “is embedded in the politics of research methods” (O’Cathain 2010, p. 532) actually prevents researchers from aligning themselves with either a quantitative or qualitative methodology—which, it has to be said, is not in itself a bad thing. Other authors have used the existing language of quantitative and qualitative research to refer to the quality of mixed methods research. Accordingly, the terms ‘construct validity’ (Dellinger and Leech 2007), ‘validation’ (Giddings and Grant 2009), ‘validity’ (Collins 2015), ‘rigour’ (Onwuegbuzie and Corrigan 2014), and ‘quality’ (Curry and Nunez-Smith 2015) have been suggested to refer to quality in this type of inquiry.

While this linguistic diversity is indicative of the openness and flexibility of the mixed methods field (O’Cathain 2010), it may actually hinder researchers’ day-to-day practice in quality assessment of mixed methods studies in two ways. First, the heterogeneity of terms used may confuse some researchers, especially those less knowledgeable of the literature on mixed methods, or researchers from disciplines where mixed methods are less established. If, as is currently the case, the literature suggests a range of terms to essentially describe the single concept of quality, then this plurality in language might well leave researchers feeling unsure and discourage them from appraising quality in their respective mixed methods studies. The second problem lies in the confusion that results from terms being often used as synonyms in the same publication—as happens in Ihantola and Kihn (2011), where the terms ‘quality’ and ‘validity’ are used interchangeably, and in Brown et al. (2015), where the same happens with ‘quality’ and ‘rigour’. Therefore, in the interest of promoting coherence and facilitating application, we endorse calls by Morse and Niehaus (2009) and O’Cathain (2010) for greater consistency by mixed methods scholars in their use of quality-related terminology.

3.3.3 Third recommendation: reach an agreement on core quality criteria

Whether or not a consensus regarding quality criteria for appraising mixed methods studies is desirable is another point of contention. Reaching an agreement on criteria is a major challenge in the field, especially given (as mentioned previously) the significant differences in the ways authors approach quality. This challenge looms even larger when we consider that there is little agreement on quality criteria even for qualitative research (Sandelowski and Barroso 2007; Seale 1999).

Three positions on the issue of consensus were identified in this review. A first group of authors (Ivankova 2014; Morse and Niehaus 2009; O’Cathain 2010; Pluye et al. 2009) consider that developing a consensus is necessary to promote clarity and to give structure to the field. Of these authors, O’Cathain (2010) suggests that a consensus could be reached through empirical expert agreement exercises; in contrast, Papadimitriou et al. (2013) argue that consensuses are developed over time through peer-review processes whereby “researchers share information and ‘educate’ each other in the co-construction of quality” (p. 149).

Other mixed methods scholars (Cheek 2015; Collins et al. 2012) consider that a consensus is neither desirable nor feasible, given that quality is heavily context-dependent and that views on what good mixed methods research depend on two key factors: the attributes of the researcher and the contingencies of each study. Accordingly, researchers from different disciplinary, philosophical, and theoretical backgrounds will inevitably have different perspectives on quality. Additionally, each research design needs to be judged according to the underlying assumptions and potential threats to quality. Therefore, any attempt to reach a consensus regarding quality would result in the following problems. First, the criteria would presumably be irrelevant for researchers who did not share the perspectives and values implicit in the proposed criteria and consequently, would be inapplicable to certain studies. Second, in contrast to the emerging and multidimensional nature of mixed methods research, the obligation to use an agreed and ‘correct’ set of criteria applicable to multiple research instances may stifle innovation and creativity. Finally, the development of a consensus may hide several dimensions of power and status associated with the politics of social research; it could well be the case that the consensus derives from a limited number of agents with the capacity to define the parameters of the field, thereby ignoring other groups “who are unable to influence the discourses of social research” (Seale, 1999, p. 29). On that specific point, Cheek (2015) reflects on the recent attempts by the US National Institutes of Health (NIH) and the Journal of Mixed Methods Research to define best practices for the field and to determine the conditions for funding a research project or publishing a manuscript. This author argues that, although the establishment of such criteria is certainly reasonable, it may lead to an endorsement of certain ways of understanding the quality of mixed methods research while excluding others.

Finally, Creswell and Plano-Clark (2011), Bryman (2014), and Mertens et al. (2016) opt for an intermediate position. In order to ensure that quality criteria are effectively used by the mixed methods community, they consider that minimum agreement is necessary regarding what these criteria are in the first place. However, these authors also recognise that achieving such an agreement is complicated, due to the fact that quality is shaped by the context of each researcher and study. To overcome this challenge, they propose that researchers agree on a parsimonious set of core criteria that are broad and flexible enough to accommodate different researcher and study contexts.

There are strong arguments in favour of this third position as probably the most appropriate way to address the quality consensus debate. It also provides an answer to the dilemma posed by Hesse-Biber and Johnson (2013): how to strike a balance between quality and diversity within the field. By agreeing on core rather than specific criteria, the mixed methods community is more likely to adopt a unified voice while, at the same time, respecting the complexity and individual viewpoints inherent to the field. A brief and simple list of core criteria would also be easier to understand and digest by emerging scholars. Yet another argument in favour of this position is that, as reported previously in Sect. 3.2, most of the core quality criteria suggested by Bryman (2014) are already shared—despite differences in authors’ perspectives—across existing quality frameworks and publications, as examined in this review.

4 Conclusion

This article contributes to the field of mixed methods research by updating previous reviews on the issue of quality. It also goes a step further in examining current trends in the literature on the quality of mixed methods research and in identifying a number of opportunities to advance the field. We demonstrate that the interest in the quality of mixed methods has grown considerably in recent years, not only quantitatively (in terms of the number of publications) but also with regards to the maturity of discussions. Authors from multiple disciplines and countries are contributing to the debate on how to critically appraise mixed methods studies, with the consensus largely being that quality criteria need to be established. We identified a number of shared core criteria across publications, despite differences in authors’ perspectives.

Finally, we propose three recommendations to guide a future agenda on the quality of mixed methods research, namely, to increase the number of empirical publications on quality, to achieve greater consistency in quality terminology, and to reach a consensus of a set of core quality criteria. Far from constraining the field, it is expected that these recommendations will open up new avenues of conversation among mixed methods scholars on how to efficiently, and respectfully, progress on this topic.