Results from a large-scale national study of more than 3000 researchers from a wide array of disciplines supported practices pertaining to open sharing and evaluation of research findings—even though many also expressed feelings that most researchers today deviate from such ideals in practice (Anderson, Martinson, & De Vries, 2007). These ideals are reflected by the recent upsurge in the development and promotion of open science practices, which refer to the openness, integrity, and reproducibility of research findings and materials (Grand et al., 2017; Nosek et al., 2015). Examples of open science practices include making study materials freely accessible (e.g., data, measures, experimental protocols, and analysis files), pre-registering study designs (i.e., registering a study and analysis plan prior to data collection), and offering open access to journal content.

In response to concerns about a “reproducibility crisis” (Baker, 2016), the open science movement and its associated practices are being discussed fervently within scholarly circles (Antonakis, 2017; Bosco, Aguinis, Field, Pierce, & Dalton, 2016; Grand, Rogelberg, Banks, Landis, & Tonidandel, in press; Hollenbeck & Wright, 2017; Ioannidis, 2005; Simmons, Nelson, & Simonsohn, 2011) as well as in mainstream media outlets (Carey, 2015; Korn, 2014). Although proper implementation of open science practices should lead to marked improvements to research and practice (e.g., greater reproducibility and replicability), some open science practices have been greeted with a measure of skepticism. For example, some have suggested that data sharing can threaten the privacy of research participants (Gabriel & Wessel, 2013; Wicherts & Bakker, 2012; Wicherts, Borsboom, Kats, & Molenaar, 2006) and that new modes of scientific communication (e.g., open access journals) are not attractive or feasible, given the current business models within the publishing industry (for a discussion see Nosek, Spies, & Motyl, 2012). Furthermore, others have suggested that many open science practices such as open access to data may not be needed (Derksen & Rietzschel, 2013; Sliter, Yuan, & Boyd, 2013) and warn that certain solutions (e.g., study pre-registration) might limit the effectiveness of scientific research and/or produce unintended, negative consequences (Leavitt, 2013).

We, like many authors, have room for improvement in our implementation of open science practices—and we hope the future will be different for us and for all organizational researchers. Although open science practices are intended to bring about numerous benefits for science and its stakeholders (Nosek & Bar-Anan, 2012), their adoption rates by journals and researchers have been relatively low (Rowhani-Farid & Barnett, 2016), particularly in the organizational sciences, perhaps because of the aforementioned concerns and skepticism to which we have alluded.

Consequently, numerous questions regarding the efficacy and legitimacy of the open science movement still exist. Furthermore, although aware that open science practices exist, many researchers are often unsure how they should be implemented. Using a question-and-answer format, the current article provides answers to common questions pertaining to efficacy, legitimacy, and application of open science practices. Whereas previous reviews (e.g., Banks et al., 2016; Grand et al., in press; Kepes & McDaniel, 2013) have focused on closely related topics like questionable research practices and publication bias, a standalone treatment of open science practices is lacking, which may explain why so many questions about open science practices persist. As such, our intention is to bring focus to the open science debate by answering 18 open science-related questions and invite broader discussion among researchers regarding the merits and challenges of open science.

Overview of Open Science

Question 1: What Is Open Science?

Open science is a very broad term that refers to many different concepts, ranging from scientific philosophies and cultural norms, such as the ownership of scientific methods (i.e., communality) and the principle that scientific output should be evaluated on its merit (i.e., universalism) (Anderson, Martinson, & De Vries, 2007), to actual specific practices that operationalize such norms (Nosek et al., 2015), even as simple as consistently adhering to specific citation standards (e.g., American Psychological Association (APA) style). Other examples of open science and policies include (1) sharing data and analytic files to improve the reproducibility of research (Nosek et al., 2015), (2) redefining or explicitly justifying statistical significance thresholds to allow for more trustworthy interpretations of research findings (Benjamin et al., 2017; Lakens et al., 2017), (3) pre-registering studies and analytic plans to distinguish between confirmatory and exploratory research (Banks, O’Boyle, et al., 2016), (4) engaging in replication studies to assess the generalizability of scientific findings (Ethiraj, Gambardella, & Helfat, 2016), (5) removing pay-walls to increase access to scientific content (McKiernan et al., 2016), and (6) changing incentive systems so that researchers are rewarded for promoting an open science environment (O'Boyle, Banks, & Gonzalez-Mule, 2017).

Taken together, engaging in these and similar practices should lead to greater sharing, accountability, reproducibility, and trustworthiness of scientific materials and results (Nosek & Bar-Anan, 2012). Likewise, evidence-based management stands to benefit from these practices as practitioners will gain increased access to scientific content, which in turn could ultimately reduce the science-practice gap (Banks & McDaniel, 2011; Schmidt & Oh, 2016). Still, open science practices are a relatively new concept and, as a result, many scientific stakeholders may be unsure of their intended meaning, purpose, and utility.

Question 2: What is the Primary Purpose of Open Science Practices?

Perhaps one of the most discussed purposes of open science practices is to improve the openness, integrity, and reproducibility of research by preventing research misconduct or reducing questionable research and/or reporting practices (for reviews of the questionable research practice literature see Banks, O’Boyle, et al., 2016; Banks, Rogelberg, Woznyj, Landis, & Rupp, 2016; Bedeian, Taylor, & Miller, 2010; Kepes & McDaniel, 2013). Research misconduct occurs when scientists fabricate, falsify, or plagiarize when proposing, performing, or reviewing research, or when reporting research results (Office of Science and Technology Policy, 2000; Resnik, Neal, Raymond, & Kissling, 2015). Although the base rate for incidents of misconduct in the scientific community is very low, even one such incident can be extremely damaging to the field.

In contrast to misconduct, common examples of questionable research practices include suppressing non-significant findings and their corresponding hypotheses, presenting post hoc hypotheses and analyses that are statistically significant as if they were planned a priori (HARKing; Kerr, 1998; O'Boyle et al., 2017), as well as “cherry-picking” fit indices and/or conducting post hoc analyses (e.g., based on model modification indices) to make structural equation model results appear better than what they really are given the data (Cortina, Green, Keeler, & Vandenberg, 2017).

Pre-registration of studies and open sharing of data, through platforms like the Open Science Framework (OSF) or aspredicted.org, can help reduce the prevalence of questionable research practices, even those where well-intentioned authors are HARKing (Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012), or when well-intentioned reviewers suggest removing non-significant hypotheses from a paper, which leads to publication bias (Banks, Kepes, & McDaniel, 2015; Kepes, Banks, McDaniel, & Whetzel, 2012). Researchers have advocated and engaged in open science practices, such as sharing data and R or SPSS scripts for data analysis, largely in the attempt to increase transparency and prevent or reduce the frequency of questionable research practices (Banks, Rogelberg, et al., 2016; Kepes & McDaniel, 2013; Nosek et al., 2015; O'Boyle et al., 2017; Wicherts et al., 2006).

Question 3: How Effective Are Open Science Practices in Eliminating Engagement in Questionable Research Practices?

As many open science practices are relatively new, it is worth noting that there is limited evidence regarding their effectiveness in reducing questionable research practices (for a systematic review see Banks, Rogelberg, et al., 2016). However, we do know that openly sharing data can lead to a reduction in sample-level publication bias as well as outcome-reporting bias (Banks et al., 2015; Kepes et al., 2012). Study pre-registration can also prevent publication bias (Kepes & McDaniel, 2013) as well as hypothesizing after results are known (HARKing; Wagenmakers et al., 2012). New manuscript submission formats such as result-blind reviews can help to reduce the prevalence of biased results in the published literature (Findley, Jensen, Malesky, & Pepinsky, 2016; Grand et al., in press). Yet, much more empirical research that evaluates the effectiveness of specific open science practice for addressing specific questionable research practice is needed (see also question 18 in this article).

Open science practices are more likely to reduce those questionable research practices committed by well-intentioned researchers; they will never eliminate the nefarious behaviors of nefarious people. What’s more, such practices could open up new categories of questionable research practices. For example, one might “pre-register” a study that has already been completed to ensure that the findings are attention-getting, statistically significant, and with no appearance of p-hacking. Unfortunately, so long as such findings are highly valued and remain a measure of scientific success (e.g., jobs, promotions, awards; Banks & O’Boyle Jr., 2013), at least some researchers will continue to pursue those values dishonestly.

Question 4: Besides Reducing Engagement in Questionable Research Practices, What Other Benefits Exist for Open Science Practices?

Open science practices have notable benefits in addition to reducing questionable research practices (Schwab & Starbuck, 2017). First, open science can promote more collaboration (Fang & Casadevall, 2015). For instance, the sharing of data may facilitate greater communication between researchers with similar interests. It may also produce meta-analytic reviews that are more useful and effective, such as item-level meta-analyses that rely upon raw data (e.g., Carpenter, Son, Harris, Alexander, & Horner, 2016). The use of digital object identifiers (DOI) will allow researchers to be assigned appropriate credit for sharing their data.

Second, the sharing of design protocols, measures, and analytic scripts can help to improve the rigor of study designs (Nosek et al., 2015) as well as reproducibility and replication rates of success (Open Science Collaboration, 2015; Schmidt & Oh, 2016). Analytic scripts should enhance the validity of the results presented by ensuring that the right analyses were used correctly to test the research hypotheses. Additionally, these shared resources can be cited, which provides researchers with a greater opportunity to receive credit for their intellectual contributions (Nosek et al., 2015).

Third, open science practices, taken together, may facilitate a better understanding, review, and improvement of the scientific process. Often, traditional manuscripts present a highly streamlined version of the research process, such that many important judgment calls are not reported. By contrast, pre-registration materials provide all scientific stakeholders a better understanding of how to continue to improve and modify designs and measures (Nosek et al., 2012). Fourth, opening scientific communication via open-access publishing could lead to faster and more widespread dissemination of research findings (similar to what has happened with ArXiv and PsyArXiv, which are open e-print archive for thousands of articles in physics, mathematics, psychology, and computer science). Currently, pay-wall systems reduce access to scientific results that can be used to inform evidence-based management and, thus, not only widens the science-practice gap (Banks et al., 2016) but also acts as a barrier to those who wish to tackle it. Online repositories that provide open access to journal content provide a means to overcome this barrier and can help avoid the suppression of studies with null effects, which will aid in meta-analytic research (Kepes & McDaniel, 2013) and a clearer understanding of our science, overall. In sum, open science may lead to both increased quality and credibility in our research, in part not only due to a reduction in certain questionable research practices (Bedeian et al., 2010; O'Boyle et al., 2017) but also due to a more positive and productive research culture and greater sharing and understanding of the scientific process behind published results (Schwab & Starbuck, 2017).

Open Science and Null Hypothesis Significance Testing

Question 5: Why Do So Many Open Science Practices Center on Problems with Null Hypothesis Significance Testing?

Null hypothesis significance testing (NHST) is a deeply ingrained paradigm within many fields of research, and within that paradigm, statistical significance has, for decades, been the primary measure of the worth of scientific findings (Lykken, 1968). Yet of course, a statistically significant finding does not mean that the theory and methods were appropriate; and conversely, a non-statistically significant finding does not necessarily reflect flawed theory or methodology.

Proper application of NHST does not inherently lend itself to a lack of transparency any more than other statistical techniques, yet transparency issues arise in most of our research that involves NHST (O'Boyle et al., 2017; Schmidt & Hunter, 2015). For example, researchers may choose to “p-hack” by excluding data post hoc as a means to convert a statistically non-significant result into a statistically significant one (Banks, O’Boyle, et al., 2016; Bedeian et al., 2010), yet we do not see researchers engaged in this practice; we only infer it after the fact. In addition, researchers may selectively include control variables that render a statistically significant result (John, Loewenstein, & Prelec, 2012; O'Boyle et al., 2017); and likewise, we may only understand this after the fact, if at all. Researchers may also engage in optional stopping, where they peek at their data as they get collected, and stop data collection once NHST yields statistically significant findings. These practices can perpetuate the belief that NHST motivates questionable research practices and therefore might be at odds with open science.

Many researchers operate under the assumption that rejecting the null hypothesis is key to publication (Sterling, Rosenbaum, & Weinkam, 1995). To the extent that gatekeepers are inappropriately using p values as a proxy for research quality, then the reward structures (e.g., publications, placements, tenure) incentivize statistically significant results rather than theoretical logic and rigorous, appropriate research designs. By emphasizing the importance of the ends, the means or process of how research is conducted and shared are diminished (Grand et al., in press). It is emphasizing the latter that is critical for open science practices.

Importantly, it is not NHST itself that necessarily runs counter to open science principles; rather, it is the ways in which researchers conceal aspects of their research conduct that can ultimately invalidate the approach. Data mining, post hoc model re-specification with cross-validation, and exploratory findings can all be perfectly appropriate and yield important discoveries when couched in the appropriate context of discovery and the future need to replicate (Jebb, Parrigon, & Woo, in press). However, ignoring or misrepresenting the underlying process of discovery violates the central open science tenet of transparency.

Question 6: How Do We Align Current NHST Techniques with Open Science Practices?

Researchers, reviewers, and editors/journals can take several key steps to align current NHST practices with open science. Important steps include (but are not limited to) altering journal policy regarding data sharing (O'Boyle et al., 2017), educating reviewers on (in)appropriate suggested edits (Cortina, 2015), and encouraging editors to solicit inductively oriented research (Woo, O'Boyle, & Spector, in press). Regarding this latter point, there is a continuum of models that can be tested that range from highly inductive or speculative (e.g., two-tailed tests, exploratory factor analysis) to highly deductive (e.g., based on hypothesized patterns of convergent/discriminant validity, or structural equation models) to something in between (e.g., mixed methods that integrate qualitative and quantitative methods; conducting broad planned contrasts in ANOVA or non-inferiority tests). All these models can advance science and improve our understanding. Rather than repeat these calls, we wish to instead focus on the actions that researchers can take.

First, open science advocates have recommended for years that pre-registration of hypotheses and research questions can help reduce the prevalence of HARKing (Wagenmakers et al., 2012). Compared to traditional study designs, pre-registration encourages researchers to spend more time planning their study before executing it. Specifically, pre-registration encourages researchers to present theories and their corresponding hypotheses and measures, as well as the expected boundary conditions a priori, which may help to reduce the prevalence of HARKing and p-hacking, respectively.

Second, a fundamental consideration when using NHST is statistical power; yet as evidenced by the low rate at which a priori power analysis results are reported in our top journals, little attention tends to be paid to sample size requirements prior to data collection (Bakker, van Dijk, & Wicherts, 2012; Cashen & Geiger, 2004; Maxwell, 2004), as is now required by some funding agencies (e.g., the National Institutes of Health). Open science practices can help in this endeavor, as demonstrated by Bosco, Aguinis, Singh, Field, and Pierce, (2015) who examined large open-access data repository to generate context-specific statistical power estimates

A third action is to provide the full dataset of variables tested and explored when investigating research questions of interest. Motivating this point is how researchers might have a very wide range of variables, such as those found in large archival and/or nationally representative datasets. Only some of these variables are relevant to the researchers’ a priori hypotheses, and other variables might still be used in an exploratory manner as a form of post hoc “insurance” to detect relationships or models that are statistically significant. That is, if a priori hypotheses are not confirmed, perhaps they get re-framed and re-tested post hoc with the additional data available in the dataset (i.e., the “Texas sharpshooter” approach; see Biemann, 2013). Researchers should be encouraged (when privacy concerns allow it) to share their full data set used in the current submission and make a very clear distinction between a priori hypotheses and post hoc exploration. So long as one is transparent, then it is fine to explore data. To support such conduct, many journals now encourage that a data transparency table be submitted with all new submissions, listing all variables contained in a dataset, and all research projects/papers using/reporting on the data (see http://www.apa.org/pubs/journals/apl/data-transparency-appendix-example.aspx).

Question 7: Will Other Analytic Approaches Solve the “Questionable Research Practice Problem” and Reduce the Need for Open Science?

A transition from the use of NHST to a Bayesian approach, a machine learning approach, or any other approach might help to address some questionable research practices (e.g., machine learning focuses on cross-validation to avoid findings that capitalize on chance). However, new problems will arise (e.g., predictive successes from machine learning can be impossible to interpret substantively), and there will always be the motivation for researchers to rose-tint one’s findings. For instance, one could move away from the statistical significance of p values to the practical significance reflected in Bayes factors, and yet Bayes factors can be “hacked” as well (Banks, O’Boyle, et al., 2016), and they are often very highly related to p values in published research (see Fig. 3 of Wetzels et al., 2011).

Regardless of whether researchers use Bayesian or frequentist approaches, the major issue is the transparency and accuracy of data and results, both of which can be threatened within any framework of research practices. As another example, if we were all qualitative researchers, open science would still be needed to help judge whether researchers are truly allowing themes to emerge from data where they exist rather than imposing themes onto the data that do not exist (Banks, O’Boyle Jr., et al., 2016; O'Boyle et al., 2017). All acceptable methodologies and statistical techniques are viable under the open science paradigm, so long as they are conducted and reported appropriately and transparently.

Applications of Specific Open Science Practices

Question 8: What Are the Differences Between Study Pre-Registration, Registered Reports, and Result-Blind Reviews?

Study pre-registration occurs when a researcher independently registers the research questions, hypotheses, design, and analysis plan via an independent organization (e.g., Center for Open Science). In the social sciences, this pre-registration (see https://cos.io/prereg/) is typically not public (hence, researchers need not fear for having their ideas stolen or “scooped”), and researchers may include an anonymous link in a manuscript submission to allow for the typical blind peer-review. Conversely, in a registered report (see https://cos.io/rr/), a researcher submits a journal proposal (e.g., introduction and detailed methods sections). After a revise and re-submit process, the journal may grant an in-principle acceptance, indicating that the study will be published provided the researchers complete the study as described in the proposal. Finally, in a result-blind review (see LeBreton, 2016), a researcher submits a study to a journal that has already been completed. However, reviewers do not have access to the results and discussion sections. Consequently, reviewer comments are focused on the theoretical and/or practical contribution of the work as well as the methodological rigor without being potentially biased by the results of the study. Thus, the reviews are more scientifically valid and follow more closely to the standards of a robust scientific discipline (Grant et al., in press).

Question 9: To What Extent Does Pre-Registration of Research Studies Decrease Creativity, Flexibility, and Prevent Serendipitous Findings?

It has been suggested that open science practices, such as study pre-registration or related practices (e.g., a priori registries and pre-data reviews of manuscripts) may prevent spontaneous discovery of scientific findings (see Leavitt, 2013). Underlying this claim is the idea that editors and reviewers may not always be supportive of researchers who explore their data (Locke, 2007; Spector, Rogelberg, Ryan, Schmitt, & Zedeck, 2014). There is truth to the notion that if researchers pre-register study designs and analysis plans, reviewers and editors will be able to distinguish between a priori and post hoc analyses more clearly. Consequently, authors may feel that any exploration of their data might be rightfully disclosed, yet be concerned that reviewers will not be supportive of exploratory quantitative and qualitative analyses.

We should work to alleviate this concern. Anyone familiar with the research process knows that, within studies and across studies, research is both theory driven and open to exploration, and sometimes the distinction unfolds as the research is conducted, not beforehand. To give just three examples: (1) before a study is conducted, an organization might provide a researcher with the names of the variables that were measured, but later not provide access to specific item content or data underlying the scores for those variables; (2) samples and populations may not be well defined before a study is planned, perhaps because department and employee participation is uncertain until the study is conducted; and (3) new opportunities arise from data that could never have been anticipated early in the process (e.g., news arises that additional survey data will be collected combined with the original data plan). Authors, editors, and reviewers should struggle with this reality. Authors should feel comfortable portraying an imperfect-yet-honest reality (Hollenbeck & Wright, 2017), and they should make use of online supplemental materials when the details are numerous. Editors and reviewers should remain critical of research under review, yet not discourage honest and transparent portrayals.

Question 10: What Are the Benefits of Study Pre-Registration?

In the context of scientific research, pre-registration is in many ways a synonym for planning. When pre-registering a project, a researcher plans and shares all knowledge regarding a study to be conducted (e.g., a summary of prior theory and current hypotheses, along with associated measures, manipulations, samples or sampling plan, and analyses). Pre-registration is something like a grant proposal—and like a grant proposal, a pre-registered study could be subject to scrutiny by colleagues and other experts, who can recommend improvements before resources are expended on the study itself. This is perhaps the most important and most underrated contribution of pre-registration.

A second benefit to pre-registration lies in transparently disclosing and drawing a clear line between the confirmatory versus exploratory aspects of research (Kepes & McDaniel, 2013; Simmons et al., 2011). Both confirmatory and exploratory work might be specified in pre-registration; however, the researcher is obligated to specify any additional changes in design and analyses after pre-registration as exploratory. Consequently, pre-registration helps researchers to protect themselves against hindsight and confirmation biases (Antonakis, 2017; Nuzzo, 2015; Wagenmakers & Dutilh, 2016).

A third benefit is that editors and reviewers would hopefully become more receptive to findings considered more exploratory or serendipitous in nature, given an improved and more rigorous understanding of the entire context and conduct of the research (Hollenbeck & Wright, 2017). Clearly, inductive research can lead to promising theoretical advances and new lines for deductive research (for famous examples see Bandura, 2001; Locke, 2007). Overall then, pre-registration of studies has great benefits and should be encouraged.

Question 11: What Are the Benefits of Sharing Data?

In today’s age of the internet, data sharing and archiving seem to be absolutely necessary activities that further our science (and also the sharing of analytic code). Reasons for losing data include human error (i.e., failure to properly store the original data) and software or hardware obsolescence (i.e., data were stored on a format and/or system that can no longer be accessed). And indeed, evidence suggests that data that are not shared are lost at an alarming rate across scientific disciplines (Wicherts, 2016), which strongly suggests storing data online for preservation purposes. Unfortunately, data loss can produce gaps in scientific literatures that threaten the efficacy of meta-analyses and other statistical procedures used to build cumulative scientific knowledge and inform evidence-based practice.

To tackle this problem, a strong culture of data sharing can be developed, with the goal of reducing a variety of questionable research practices (Tenopir et al., 2011), also yielding down-stream benefits for future research that can make use of and otherwise be inspired by shared data (Wicherts & Bakker, 2012). Furthermore, as mentioned previously, journals that attach a DOI to a shared data set can increase visibility/citation of the contributing author (see http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0000308).

Question 12: Why Might Researchers Be Hesitant to Share Data?

Researchers and other scientific stakeholders may have several reasons for not sharing research data or supporting such sharing. Data can be very difficult to collect; they are inspired by the researchers’ own ideas, and they are a form of wealth and competitive advantage in the research community. All of these factors may make a researcher feel that sharing data would mean “giving away” data (Savage & Vickers, 2009). There are more arguably legitimate reasons as well for not sharing data. First, the sharing of some types of data could potentially compromise the identities of certain participants (for a discussion see Gabriel & Wessel, 2013). For example, collecting birthdate, gender, and zip code information is enough to identify 87% of U.S. residents (Tanner, 2013). Essentially, data thought to be anonymized can sometimes be paired with other existing data to reveal the identity of participants. The likelihood of this happening only increases in a world of big data. Additionally, researchers can err and fail to redact identifying information (e.g., company name mentioned in an open-ended comment). Thus, there is some risk when sharing data.

Second, organizations may be less likely to share proprietary information if data must be shared with third parties (Wicherts et al., 2006), especially given the concerns about employees’ or the firm’s information being compromised (Jones & Dages, 2013), as well as the restrictions of domestic and international privacy laws (e.g., HIPPA, FERPA, GDPR; Privacy Shield). And just like researchers, organizations also may not want to give up a competitive edge from a data set in which they invested great insight and effort. Researchers are, therefore, often required to sign non-disclosure agreements in exchange for being able to analyze and interpret organizational data. If a research endeavor requires complete transparency and data sharing, then organizations may feel well justified in choosing not to participate in research. For example, data sharing could create legal liability issues as freely available data that can be traced back to an organization could be used as evidence in lawsuits. Even fears of having data subpoenaed are often sufficient for legal departments to decline research proposals focused on diversity. This higher standard threatens to make getting access to field data even more difficult for diversity scholars for this reason.

Another stream of concern comes in the form of researchers worrying about others discovering flaws in their research (Nosek et al., 2012). Criticism is the very core of scientific discourse and progress, yet criticism can also lead to negative inferences about a researcher’s reputation or credibility. In addition, researchers may be concerned that in future research, their shared data will be misused or used in unintended ways (Tenopir et al., 2011). Finally, editors and publishers may be hesitant to impose new requirements around data sharing because of inertia, concerns about journal impact factors, and concerns about increased workload/lack of resources to support such policies (e.g., the need to manage a data sharing process). In summary, there is truth to the claims that not all data can be shared, that there could be negative consequences to sharing data. To be clear though, that is not to say that we cannot improve our data-sharing policies in the research, organizational, and publishing communities. We can.

Question 13: How Can the Challenges of Data Sharing Be Addressed?

Many data sharing concerns could be addressed through careful policy implementation and enforcement. For example, the default for data sharing may be opt-in, although authors could be given the option to provide the journal editor with legitimate reasons for not sharing data. As a compromise, upon acceptance of the article, journals might allow authors to request an embargo window of time, ranging from a few months to several years during which data are not shared, giving authors ample time to re-use the data if desired and appropriate. Requiring authors to de-identify and annotate data carefully before sharing might also quell concerns about data misuse by future researchers (Wicherts, Bakker, & Molenaar, 2011). Careful data annotation and the knowledge that data will be shared could also improve the quality of the original work being carried out, reducing the frequency of future corrigenda and retractions, subsequently improving the reproducibility of our science (Nosek et al., 2015). In short, one should not jump to the conclusion that all data—or even any particular data set—can or cannot be shared. Careful consideration is generally required in light of the joint concerns of open science, editorial policies, and the nature of the data and study themselves.

Currently, many journals do not have a well-articulated written policy regarding their data sharing policies, yet they should (Banks, O’Boyle, et al., 2016; Banks, Rogelberg, et al., 2016; Nosek et al., 2015). Often, journals simply require that researchers follow standards put forth by scientific and professional societies such as the APA’s Publication Standards, which requires authors to make their “data available to permit other qualified professionals to confirm the analyses and results … for a minimum of five years after publication of the research” (American Psychological Association, 2010, p. 12). Journals affiliated with the APA have authors sign a contract that they will make their data available to peers to verify the findings (Wicherts et al., 2011). Yet in one notable study, 141 psychology researchers were approached for data published in APA outlets less than 12 months earlier. Although 27% shared at least some of their data, the remainder failed to comply with the request (Wicherts et al., 2006). Consequently, even journals promoting data sharing requirements currently have problems with compliance. Not only should journals have formal policies that are enforced; they should generally encourage and facilitate authors’ willingness to share their data (even during submission so that reviewers can verify analyses as helpful; this is very rarely done). For example, journals could provide a user-friendly interface for authors to upload their data when submitting their original manuscript (e.g., see https://www.aeaweb.org/journals/policies/data-availability-policy).

Finally, we suggest that institutional training on responsible conduct of research (RCR) focus more explicitly on privacy concerns, including data privacy. RCR training, which is provided at all U.S. research institutions and required by federal funding agencies, could focus on teaching de-identification and aggregation methods so that researchers can better protect participant identities, making it easier to facilitate open science through data sharing. Authors that could never share the raw data, at a minimum, could be trained to use de-identification and aggregation methods that might still allow for useful data sharing. For instance, an organization might prohibit the sharing of individual-level data, but all information required to reproduce analyses could still be provided, such as descriptive statistics, intercorrelations, and reliability estimates. Statistics at the item level and/or subgroup level might also be provided as online supplementary material, as necessary to reproduce all analyses.

Question 14: How Does Open Access Publishing Affect the Activities of Publishers and Other Stakeholders?

The shift towards an open science model is already causing an economic change to the business model that exists between publishers and consumers of science (e.g., researchers, practitioners, university libraries, professional associations and societies; Nosek et al., 2012). Yet publishers remain concerned about losing financial control, should the shift become seismic. Indeed, the science-publishing industry generated approximately $12.7 billion in revenue in 2014 (Healy, 2015), and journal subscriptions are often considered a major source of revenue for many publishers and professional associations that own or sponsor journals. For example, publication sales earned the APA and Academy of Management (AOM) $13,662,191 and $3,063,708, respectively, in 2014 (Internal Revenue Service, 2014). Importantly, these numbers represented 10.5 and 25.4% of each association’s respective total revenue for that year. Many of the critical activities and services provided by these professional societies, including the professional development of members, are funded in part by this revenue.

Similar to the music industry, the open science movement could have a negative impact on these revenue streams (Nosek et al., 2012), unless greater revenue-generating innovation takes place. For example, to maintain their operations, journals may levy an open access charge to authors to have their research published. This may be acceptable to researchers who budgeted for such expenses in their grant proposal; however, open access charges may discriminate against students, those unaffiliated with universities, and those academic departments that are less well funded. Furthermore, printing scientific articles is just one of at least 96 things that journal publishers do (see Anderson, 2016). In addition to publishing, they manage and protect subscriber records, engage in DOI registration and search engine marketing, as well as maintain e-commerce systems, copy editing, plagiarism checks, respond to legal actions, and engage in product marketing and market research. Revenue generated from subscriptions and publication sales likely support many of these functions. Under an open science model, these revenues may dwindle or disappear, which may threaten a publisher’s ability to offer its customers the same level of service.

All of this being said, many open science practices may end up benefiting journals, publishers, and professional associations. For example, open-access journals may be more marketable, useful, and appealing to a wider range of practitioners than pay-walled ones. Open-access journals may create a much larger footprint of access, visibility, and loyalty through their offerings.

Future Directions for Open Science Research

Question 15: What Are the Primary Shortcomings of Open Science Practices?

Recommendations in favor of open science (within both the popular press and the academic literature) can be very convincing (e.g., Banks, Rogelberg, et al., 2016; Nosek et al., 2015). Indeed, open science practices offer prime opportunities for all relevant stakeholders to improve scientific conduct and communication. Open science focuses on the process of research: e.g., being more transparent, pre-registering studies, and sharing data. Yet, open science is not a panacea and cannot address all of the problems inherent to contemporary research.

First, open science alone does not fully address rigor or relevance issues (Vermeulen, 2005). Much of the emphasis in open science is on full and transparent explanation of research methodology and the sharing of all observed data. Although these practices can improve the ability of other researchers to understand and even replicate work, they can only partially address the quality of the original work. To the extent that research makes use of insufficient or suboptimal methodology (e.g., biased sampling, reliance on single-source data to test complex relationships, using cross-sectional data to test causal predictions), open science practices to date will not substantially improve the quality of what appears in the literature. In other words, open science is not always better science, although multilab research and input from pre-registration may improve our science. Of equal note, open science efforts do not directly speak to what we study. Specifically, open science is not always useful science. That is, open science does not directly address the issue that, ideally, researchers should invest their time and insights into lines of inquiry that take risks, yet, on the whole, stand a reasonable chance of meaningfully contributing to collective knowledge in a way that informs practice and betters society (Banks, Pollack, et al., 2016).

Second, and closely related to the first point, open science alone does not directly address or improve statistical power. Much of our research is known to be underpowered (Bakker & Wicherts, 2014; Maxwell, 2004; O'Boyle, Banks, Carter, Walter, & Yuan, 2018). Pre-registration and grant requirements for statistical power can address this issue, as does research that seeks to replicate underpowered studies more robustly with larger samples in a series of studies (e.g., Donnellan, Lucas, & Cesario, 2015). Third, there are legitimate concerns about feasibility and implementation of certain open science practices. As one example, implementation of a registered reports model into our publication processes is neither easy nor resource-free. Suggesting an alternate path to publication that deviates from a journal’s typical practices can be met with resistance from publishers, editors, reviewers, and authors, due to various costs (e.g., people dedicated to implementing change; time in setting up new policies; money required to change manuscript-processing systems; cognitive and motivational resources in educating those involved and dealing with resistance or other consequences). There also may be the concern that high levels of investment in open science should not be made until there is a large scale well-established open science culture that assures a return on that investment. But this is an obvious chicken-and-egg problem for the field. Compounding this problem, editors of prestigious journals with high impact factors may have few, if any, incentives for changing their systems, with both real and perceived disincentives for doing so.

Fourth, the law of unintended consequences suggests that altering the system through which we conduct and share our knowledge will lead to other problems and perverse incentives. For example, if we encourage researchers to publish their work following different paths, some individuals and institutions may be put at a disadvantage. Journals with high impact factors may not embrace open science as quickly, yet may likely continue to be viewed as a “gold standard,” whether it is for tenure and promotion decisions, or for the ranking of colleges and universities on national research productivity. Thus, encouraging graduate students and junior researchers to take an alternate path to sharing their work might almost inevitably mean that their placement and tenure chances are diminished, unless the open science culture is an institutional and professional one, not merely residing within journals, where high-quality publications are important and cited wherever they appear. But without “elite” journals serving as a signal, it is harder for a consumer and perhaps even researchers themselves to discern or vet high-quality research quickly. On the other hand, we know that there is more good research produced than can be held in our elite journals. We also know that excessive reliance on statistical significance and p-hacking even in our elite journals has had negative consequences for our science. And finally, citation rates and the name of the journal in which a paper was published are useful but highly imperfect indicators of research quality; we should continue to seek out better methods for identifying and reward high-quality scholarship (Nosek et al., 2012).

Question 16: What Are Some Challenges for Adopting Open Science Practices?

Although there are numerous answers to this question, two deserve comment. First, the open science provider (i.e., the individual or entity engaging in open science) is faced with many technical barriers that likely further attenuate community-level participation (Janssen, Charalabidis, & Zuiderwijk, 2012). Perhaps the greatest bilateral technical barrier facing both the providers and users of open science practices is the lack of a supporting infrastructure (Janssen et al., 2012). Such an infrastructure might be linked to journal publisher webpages and allow researchers to make available their datasets, analytic scripts, and documentation detailing important decision rules for their study. The OSF, which provides an online platform where researchers can create project pages and make available the corresponding study materials (e.g., datasets, analytic scripts), is working hard to break down this barrier. Despite these efforts and substantial outreach efforts on the part of the OSF, many researchers are unaware that this open-source architecture exists—or how it can benefit their research, their reputations, and the scientific record in general. Significant graduate education and training will likely be required before open science practices become the behavioral norm in many fields of research.

Second, it is important to consider what legal ramifications might follow a shift to an open science model and how open science practices might affect policy change (Friesike, Widenmayer, Gassmann, & Schildhauer, 2015) and the commercialization of intellectual property (Caulfield, Harmon, & Joly, 2012). At the same time, however, public outcry following highly publicized incidents of research misconduct (e.g., see Bhattacharjee, 2013) has led some government agencies to take action against scientists found to have engaged in research misconduct (McCook, 2016). We note, however, that federal policy is slow to develop and change, whereas bottom-up change can be quick and disruptive.

Question 17: What Major Steps Have Been Taken to Promote Open Science Practices?

Although many challenges surround the open science movement, a number of key advancements have already been accomplished in some scientific areas (Nosek & Bar-Anan, 2012; Nosek et al., 2012). Recent initiatives include, for example, the aforementioned Center for Open Science and its pre-registration challenge (https://cos.io/prereg/). The primary objective of this initiative is for researchers to distinguish confirmatory and exploratory analyses in order to retain the validity of their statistical inferences. A second example is the Editor’s Code of Ethics (http://editorethics.uncc.edu/), which represents a set of standards that are intended to have a positive impact on the way journal editors conduct themselves as well as the quality and integrity of research. Relatedly, the Committee of Publication Ethics (COPE, http://publicationethics.org/) provides a forum that offers advice to editors and publishers on how to handle cases of research misconduct.

In addition, the Transparency and Openness Promotion (TOP) guidelines (Nosek et al., 2015) are a set of modular standards that can be adopted by journals in whole or in part as a means to move scientific communication towards greater openness. At the time of this writing, more than 5000 journals are signatories of the TOP guidelines as well as three of the four biggest publishers (Elsevier, Springer-Nature, and Wiley). There also exists data and other quantitative depositories, such as metaBUS (http://metaBUS.org; Bosco, Aguinis, Singh, Field, and Pierce, 2015,b), which can be used to centralize scientific findings and metadata at the effect size level to facilitate literature searches and meta-analyses. Finally, the Open Science Grid (http://opensciencegrid.org/; Pordes et al., 2007) is a multidisciplinary partnership that provides high throughput computing for research in the USA. In 2016, it provided 1.2 billion central processing unit (CPU) hours to researchers across a wide variety of projects.

Moreover, recent evidence suggests that the pervasiveness of open science practices may be increasing (Munafò et al., 2017). For instance, some journals within the social sciences now provide alternative paths to publication (see LeBreton (2016); https://osf.io/8mpji/wiki/home/). Other journals offer rewards (e.g., badges) that recognize those who engage in open science practices (Eich, 2014; Grahe, 2014). Furthermore, since 2008, the worldwide share of published articles that are open access has grown at a consistent rate both relatively speaking and in absolute numbers (Butler, 2016).

Question 18: Where Should Future Research Related to Open Science Practices Be Directed?

What open science efforts are worth prioritizing and investing in, given the limited time and resources available? Future empirical research on open science practices can target measures of the effectiveness of various approaches. Such open science practices and priorities will not be applied uniformly, which is actually a good thing, because the variation serves as a series of natural experiments. Open science recommendations might encourage all journals to develop a data sharing policy and at least some journals to require data sharing (with permitted exceptions). Maybe all journals will engage in some open science practices, with positive benefits to our field (Nosek et al., 2015), but our point is that these benefits can be measured in a longitudinal quasi-experimental framework across organizational journals. Open science improves research and itself can be researched via the application of its own practices.

Actionable Recommendations to Help Us Move Towards Open Science as a Field

This article provides answers to 18 questions pertaining to open science behaviors to bring a greater understanding of the issues faced in organizational research. Still, having a greater understanding in hand does not answer a critical question: “How do I engage in open science behaviors?” The following section begins to answer this question.

General Recommendations

First, we encourage researchers to visit the COS’s webpage (https://cos.io/) and become familiar with its extensive and growing list of products and services (e.g., training programs). Indeed, the advent and evolution of the open science movement have brought about numerous platforms that are intended to be aligned with the goals of the movement. For example, https://aspredicted.org/, like the COS, offers a pre-registration service, which, as described earlier, is intended to distinguish exploratory from confirmatory analyses. That said, note that the COS offers researchers an integrated approach in terms of the ability to pre-register their study, share their corresponding data files and analytic scripts, and publish pre-prints of the subsequent research product. Furthermore, the COS provides open access to a comprehensive library of training materials, which contains dozens of video tutorials on how to become a more active contributor to the open science movement (e.g., pre-registration with the OSF; see https://cos.io/our-services/training-services/cos-training-tutorials/). Taken together, we consider the COS to be the most valuable resource available to organizational researchers who seek to increase their engagement in open science practices.

Second, we suggest that researchers consider sharing at least some subset of their data files and analytic scripts to the benefit of the research community, especially those not only materials that will help to reproduce publish research findings but also materials that researchers have found invaluable to their work flow. Materials and preprints can be shared easily by creating public project webpages through the OSF platform (see https://osf.io/), which in turn allows access to other platforms like Dropbox (see https://dropbox.com) and Github (https://github.com/; Tutorial: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fosf.io%2Ftxgn8%2F&data=02%7C01%7Csctonidandel%40davidson.edu%7C82679a20f4994fe6c4ea08d59cd57d4f%7C35d8763cd2b14213b629f5df0af9e3c3%7C1%7C0%7C636587363112120519&sdata=UgX4dmmhVm%2BXx4DuW9ztZeRIQ5B5S2wCk48Pd3e8yoo%3D&reserved=0). Note that preprints can be created through an extension to the OSF platform (see https://osf.io/preprints/), which allow manuscripts to be seamlessly linked to their corresponding research materials. Furthermore, preprints are assigned DOIs, which means that they can accumulate citation counts and, thus, elevate a researcher’s visibility.

Third, we recommend that researchers develop an open science peer network, which can be initiated by reaching out to a COS Ambassador. Currently, there are more than 200 COS ambassadors who act as local information resources for the COS, OSF, research transparency, and reproducible practices. Many of these individuals have experience with the open science practices discussed in this paper (e.g., study pre-registration, data sharing) and can educate researchers on the challenges associated with each one.

More Specific Recommendations

Table 1 introduces a set of recommendations that represent challenging, yet specific, future goals for improving open science behaviors both at the individual and collective levels. In column 1, we present the target stakeholder of our recommendations. Columns 2, 3, and 4 present steps for incremental improvement in the application of open science practices following the staged approached proposed by Nosek et al. (2015).

Table 1 Actionable recommendations to implement and evaluate open science practices
Full size table

For instance, most articles do not comment on the availability of data or analytic code from the corresponding author (see “authors: step 0” in Table 1). At step 1, authors might make such a statement, or state that only aggregated data are available because of privacy reasons. At an even more progressive step (step 2), authors could upload their raw data and analytic code to an online repository (e.g., the OSF; https://osf.io/), providing an anonymized link to these resources while the article is under review.

It is important to recognize that stakeholders beyond the research community can also engage in open science behaviors and help to move the field towards greater transparency. Table 1 provides actionable recommendations for these stakeholders, such as editors of journals and other publication outlets, who often act as the bridge between scientific discovery and dissemination of knowledge and, thus, the bridge between science and practice. At step 0, editors may choose to not include a data sharing policy on their journal’s website or in their outlet’s submission guidelines. At this step, open science behaviors are not suppressed or encouraged.

A more active approach to open science involves editors developing, coordinating, and communicating policies that encourage authors to make their raw data and analytic code publicly available (see “editors: step 1” in Table 1). This recommendation represents an incremental shift towards an open science environment that all editors could enact. Indeed, an editorial policy that encourages and reminds researchers to decide mindfully whether or not they share their data and analytic code on the OSF (http://osf.io) does not require major structural and/or operational change to a journal. However, editors could decide to take an even more progressive step towards open science; one that would require an author’s data and analytic syntax be made publicly available unless ethical norms prevent it (see “editors: step 2” in Table 1). Although this is a more a radical recommendation, we are heartened by the growing number of journals and funding agencies (e.g., NSF) that now encourage or require researchers to share their research materials. This demonstrates that some of the recommendations outlined in Table 1 are possible and can be achieved over time.

In addition to these steps, Table 1 provides supplemental resources that support engagement in these behaviors. For example, we provide links to video tutorials on how to use the OSF (https://cos.io/our-services/training-services/cos-training-tutorials/), how to pre-register a study (https://cos.io/prereg), and earn badges for using open science practices (https://cos.io/our-services/open-science-badges/). The Center for Open Science also provides statistical consulting (see https://cos.io/our-services/training-services/) and a blog so that interested readers can stay up to date on the most recent open science trends (https://cos.io/blog/). Finally, we must emphasize that these recommendations will change as open science practices themselves change. As previously stated (see question 18 above), more research is needed that evaluates the effectiveness of open science practices and considers contingency factors in the effectiveness of the practices. Meanwhile, new and better open science practices are likely to emerge as the open science movement matures (Nosek et al., 2015).

We hope that the preceding discussion and our recommendations advance the discourse on open science practices, for those primarily involved with generating and disseminating research findings, as well as for those who consider and provide incentives within the research enterprise. Ultimately, open science affords us with tools to help educate, motivate, and nudge honest researchers to improve their scientific practices. To the degree that there remains motivation to conduct willful and intentional scientific misconduct, open science can only do so much to discourage wrongdoing. The open science movement instead promotes greater transparency within all stages of the research enterprise and consequently stands to benefit our scientific methods, process, and discourse.