Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

FormalPara Learning Objectives

After studying this chapter, the reader should be able to:

  • Explain when a critical review is needed and what is its purpose.

  • Provide perspectives on the difference between critical review, scientific review and validation.

  • Explain the principles, procedure, requirements, content, deliverables and options when conducting critical review and which international standards describe it.

  • Discuss the necessary qualifications of a reviewer and how they are selected and by whom.

  • Describe the possible roles, obligations, tasks, and deliverables of a reviewer.

The focus of this chapter is on understanding what a critical review is and what its main elements are. It is important to note that it is NOT the objective of this chapter to learn how to conduct a critical review, neither from a reviewer nor from a practitioner perspective.

1 Introduction

Numerous LCA studies have been published and many of them based on the highest standards of quality and robustness, but there are also an alarming number of studies that contain either important mistakes or plain manipulations in order to obtain an intended result that would support a specific, pre-defined claim. These mistakes and manipulations may be subtle and difficult to detect but can also be immediately identifiable to the trained eye and a number of studies based on surprisingly blunt and evident manipulations have been published over the years. Especially some earlier studies have become classic and illustrative examples in LCA teaching of how not to do LCA (or comparative environmental claims in general) and they also nicely illustrate the purpose and need for critical review of published LCA studies. Two entertaining examples are:

  1. (1)

    SUV versus hybrid car: A famous example is a study from the automotive marketing company CNW Marketing Research, Inc. from 2007 called “Dust to Dust: The Energy Cost of New Vehicles From Concept to Disposal”. This study compared the life cycle energy costs of a number of automobiles from 2005 and had no hesitation to conclude (and widely communicate) that many large sport utility vehicles (SUVs) including GM’s massive Hummer models H2 and H3 use less energy per mile driven than many smaller vehicles including the Toyota Prius hybrid car. Gleick (2007) analysed the information and commented “that the report’s conclusions rely on faulty methods of analysis, untenable assumptions, selective use and presentation of data, and a complete lack of peer review . Even the most cursory look reveals serious biases and flaws: the average Hummer H1 is assumed to travel 379,000 miles and last for 35 years, while the average Prius is assumed to last only 109,000 miles over less than 12 years”.

  2. (2)

    Fast-food versus classic restaurant: A study from the 1990s comparing a fast-food restaurant with a normal restaurant that surprisingly concludes the environmental superiority of the fast-food option. When the study was redone by independent practitioners, they demonstrated that the system boundaries were chosen in a way that comparability of both options was not supported since important processes from the fast-food restaurant were excluded. Correcting these manipulations then yielded a different picture (Lang et al. 1994). The whole story can be found in Jolliet et al. (2015).

Product systems can be very complex, involving a high number of processes and locations. Their modelling in LCA builds on multiple data sources from measurements to unit process databases and involves influential assumptions, drawing on a diversity of expertise from process engineering to environmental and sometimes also social science. Results are often communicated to stakeholders and decision makers that cannot control the quality of the studies, and manipulation and mistakes in LCA studies are as old as the tool itself. The understanding of a need for an independent critical review of LCA studies thus came very early in the history of the methodology. The SETAC LCA “Code of practice” proposed it first in 1993 (Consoli et al. 1993) with more detailed procedural guidelines published later by Klöpffer (1997) and Weidema (1997), which still stand until today as essential references on how to conduct a critical review . As its superseded predecessor from the late 1990s, the revised international standard ISO 14044 (2006a) defines review procedures (although in much less detail than Klöpffer and Weidema, respectively) to ensure that an LCA study is conform to ISO requirements. As a further development from there, ISO published the technical standard ISO TS 14071 (2014) that aims to specify detailed ISO requirements for critical reviews. In consequence, this should ensure that all claims of a critically reviewed LCA study are well justified and supported by assumptions, methods and data used. Besides preventing misuse and unsupported claims, critical review may also help identifying mistakes and more justifiable assumptions as well as generally improve the quality of a study. It thus supports the robustness of an LCA and increases trust in its results and conclusions .

In general, there are different kinds of review processes associated with scientific and technical developments and they all fulfil different objectives and vary in their approach and process. Two different types of review processes are mainly relevant in the context of an LCA study: (1) scientific peer-review and (2) critical review according to ISO 14044 (2006a). While this chapter is focusing on the latter, there is much confusion between both and it is essential to clearly distinguish them and understand their differences. Table 13.1 provides a simplified overview of general tendencies for similarities and differences between both types of review.

Table 13.1 Similarities and differences between scientific peer-review and critical review according to ISO 14044 (note that this table represents a general tendency for each criteria, not an absolute truth as there will likely be cases of review processes that may differ on either side of the table)
Full size table

Besides several similarities, the essential differences between these two review types are thus linked to their duration, depth, cost , transparency, confidentiality, content and objectives. As discussed by Curran and Young (2014), there is also an important and frequently ignored difference between the terms “critical review” and “verification” with their essential difference being that critical review relies on expert judgement whereas verification is based on comparison against objective evidence.

The focus of this chapter and the basis for any critical review of LCA studies are the International Standards ISO 14040 (2006b), ISO 14044 (2006a) and ISO TS 14071 (2014). However, it is worth noting that other review schemes exist that may be specified in more detail than in ISO 14044 and ISO TS 14071, while still being fundamentally based on them. These review schemes often have a very specific context of application and in most cases also a geographically limited relevance. One example is the International Reference Life Cycle Data System (ILCD) of the European Commission with its series of ILCD handbooks, including one specifically dedicated to review schemes (EC-JRC 2010a) and another to reviewer qualifications for LCI datasets (EC-JRC 2010b) linked to the European reference Life Cycle Database ELCD. While these are valuable sources of information for the interested reader and we recommend them for further study, they will not be discussed in detail in this chapter.

2 Critical Review Process

As presented, critical review is a procedure intrinsically linked to ISO 14044 (2006a) which defines it as a “process intended to ensure consistency between a life cycle assessment and the principles and requirements of the International Standards on life cycle assessment” (Clause 3.45). However, critical review may also be performed just in order to improve the quality of the study and thus the trust in it. The following will detail why, how, and when critical reviews are performed.

2.1 Purpose

Critical review of an LCA study is useful in all cases where quality, robustness , and trust in results are wanted. Whether or not a review is required depends on the goal definition, i.e. the intended application and decision context , the reasons for carrying out the study, and the intended audience. ISO 14044 recommends the use of critical reviews in general and makes it mandatory for “LCA studies where the results are intended to be used to support a comparative assertion intended to be disclosed to the public” (ISO 2006a). These mandatory critical reviews have to (“shall” in ISO terminology, which indicates an obligation) be performed by a panel of interested parties including at least three experts. A comparative assertion is defined by ISO as (ISO 2006a): an “environmental claim regarding the superiority or equivalence of one product versus a competing product that performs the same function”.

However, this definition may not be broad enough. In the European context the Product Environmental Footprint (PEF) is an example of an LCA that will typically be subject to such review requirements. Even though a comparative assertion is not explicitly stated in the report, Environmental Product Declarations (EPD) and PEF aim to give data and information to be used in comparisons and they could therefore be regarded as a basis for comparative assertions. In fact, critical review by at least one independent and qualified external reviewer (or review team) is mandatory in the PEF methodology (European Commission 2013).

2.2 Chronology

A critical review can basically be performed in two alternative ways. The first is to review the LCA after the study is completed (a posteriori review). The second approach is an integrated/interactive review where the reviewer(s) follows the study from the definition of goal and scope, through data collection to the conclusion (concurrent review). In the a posteriori approach at least one iteration of review comments and associated modifications of the study are performed and the critical review report should reflect the entire review process. In the concurrent review scheme the reviewer(s) can be involved in several of the different steps throughout the conduction of the study, i.e. (ISO 2014):

  1. (a)

    “the goal and scope definition;

  2. (b)

    inventory analysis including data collection and modelling;

  3. (c)

    impact assessment;

  4. (d)

    life cycle interpretation;

  5. (e)

    draft LCA report”

with the critical review statement being issued for the final version of the LCA report. ISO 14044/ ISO TS 14071 do not specify any requirements or preferences to one or the other approach (neither does PEF), and they can hence always be freely chosen.

Most literature recommends the concurrent review (Weidema 1997; Klöpffer 2005, 2012; Hamilton and Ayer 2013; Schulz and Mersiowsky 2013). An a posteriori critical review involves a risk of delays in the final phase. The reviewer(s) has to comment on the draft final reports usually within a few weeks and there is a risk that serious flaws in methodology or data quality , or new aspects appear. Doing the necessary corrections may be hindered by budget and timing. Also, the review process requires communication between the practitioner and the reviewer(s) and in some cases the practitioner may not be available after the completion of the study (Klöpffer 2005).

The concurrent review approach has the benefits that potential problems can be corrected at an early stage of the study. There may be some extra time needed at the beginning of the study to guide it onto the right track, but this will likely be less time consuming than delays caused by new aspects surfacing at the end of the study or by the need to figure out how assumptions and calculations influence the results. This obviously also influences the timing of the study in itself since the practitioner has to wait for review comments at different milestones throughout the study. Typically, one month additional time should be expected (Schulz and Mersiowsky 2013). A minor concern raised by Curran and Young (2014) is the risk that reviewers may become vested in the study and thus lose their independence.

2.3 Requirements

According to ISO 14044 (2006a), “the critical review process shall ensure that:

  • the methods used to carry out the LCA are consistent with the international standard;

  • the methods used to carry out the LCA are scientifically and technically valid;

  • the data used are appropriate and reasonable in relation to the goal of the study;

  • the interpretations reflect the limitations identified and the goal of the study; and

  • the study report is transparent and consistent”.

The European PEF guide (European Commission 2013) outlines the same requirements although it additionally mentions that the data quality should meet requirements and that the study report shall be accurate. This list gives guidance for the reviewer(s) and may also serve as the structure for review reports (Klöpffer 2012). In most instances the reviewer is expected not only to do an “administrative ISO check”, but to also be a discussion partner accompanying the LCA project (Curran and Young 2014).

Thus, as stated by ISO TS 14071 (2014) “The critical review should cover all aspects of an LCA, including data appropriateness and reasonability, calculation procedures, life cycle inventory, impact assessment methodologies, characterisation factors , calculated LCI and LCIA results, and interpretation”. Regarding two aspects ISO TS 14071 leaves it optional whether or not the critical review includes them:

  1. 1.

    Assessment of the life cycle inventory (LCI) model,

  2. 2.

    Assessment of individual data sets.

Curran and Young (2014) note that in contrast to the usually comprehensive review of methods and assumptions, there is often a limited examination of data and quantitative results. This may be due to a combination of limitations of time (budget), weak transparency and/or poor accessibility of data sets. In order to perform the critical review it is important that reviewers are granted access to the data and inventory model by the commissioner and practitioner.

In LCA it is difficult to establish objective quality criteria, and specific criteria for whether or not a study is correct cannot be defined. Therefore, much of the critical review has to rely on professional judgement regarding the consistency between goal and scope, data and models used, interpretations applied and the robustness of the conclusions drawn. The previous chapters in this part of the book specify in detail the requirements for conducting an LCA and thus also the aspects that reviewers should be aware of when performing a critical review. Several authors discuss in further detail the specific considerations and questions to ask during the review process, and the interested reader is referred to those (Consoli et al. 1993; Klöpffer 1997; Weidema 1997).

2.4 Deliverables

The deliverables of the critical review are (ISO 2006a, 2014):

  • Comments to specific intermediate steps of the LCA (goal and scope definition, LCI, LCIA, interpretation) for a concurrent review,

  • Comments to the final draft LCA report,

  • Review report ,

  • Review statement .

The review report documents how the critical review was conducted including all reviewer comments and recommendations given plus a response (to each comment/recommendation) from the practitioner that may indicate consequent changes applied to the study and/or the report or a justification of the respective issue in the study or the report in respect to the comment. Annex A of ISO TS 14071 (2014) contains an informative template for a critical review report. Hamilton and Ayer (2013) suggest that “In general, you should ensure that the following steps in the process are documented:

  • Review panel comments to the study team

  • Study team responses to the review panel

  • […]

  • Correspondence between the panel and the study team”.

The critical review statement is a short text that clearly states whether or not the study is conform to the requirements of ISO 14040 and 14044. It should also discuss “any particular strengths, limitations and remaining improvement potentials of the LCA study or the critical review process” (ISO 2014). ISO TS 14071 clearly states what has to be included in the critical review statement (ISO 2014):

  • “Title of the study;

  • The commissioner of the LCA study;

  • The practitioner of the LCA study;

  • The exact version of the report to which the critical review statement belongs;

  • The reviewer(s) or, in the case of a panel review, the panel members, including the identification of the panel chairperson;

  • A description of the review process, including information on:

    • Whether the review was performed based on ISO 14044:2006, 6.2 or 6.3;

    • Whether the review was performed in parallel or at the end of the study;

    • Whether the review included or excluded an assessment of the LCl model;

    • Whether the review included an analysis of individual data sets;

  • A description of how comments were provided, discussed and implemented;

  • A statement of the result of the critical review , i.e. whether the study was found to be in conformance with ISO 14040 and ISO 14044 or not”.

Hamilton and Ayer (2013) also recommend that “It is important that the final critical review statement includes:

  • The date of issuance […] of the study

  • […]

  • Documentation of any outstanding issues that were not resolved during the review

  • A summary of the comments/responses from the review process”.

The final LCA report has to mandatorily include the review statement and review report , as well as all comments and recommendations of the reviewer(s) and any responses by the practitioner to them. It is a requirement of the ISO 14044 standard that the review statement and review report must be included in the LCA report (typically as an appendix to the report). The critical review statement has to be signed by the chairperson and should also be signed by the other reviewers. This signature is strictly individual and personal and cannot be representing an institution nor be replaced by an institutional stamp or label . This means that the reviewer(s) publically (if the report is published) state the conformance or non-conformance of the study to ISO 14040 and 14044 with their names and signatures, which ensures that especially intentional manipulations (but also larger mistakes) that would affect the LCA’s conformance to ISO should have been identified and corrected. This can be seen as a sort of quality insurance, making the reviewer(s) personally responsible for the review process and content.

3 Reviewer Qualifications, Tasks and Selection

Since the purpose of a critical review is to perform a critical expert judgement as to whether the ISO 14044 criteria are fulfilled, the expert(s) should of course be independent of the LCA, but not necessarily external to the company . In fact, the foremost requirement for any reviewer, internal or external, in the context of a critical review is complete independence from the study (but not necessarily its commissioner or practitioner), i.e. not involved in the commissioner’s or the practitioner’s project team, nor otherwise implicated in the definition of the scope or the conducting of the LCA. In the case of an internal expert, this person may, however, be full-time or part-time employee of either the commissioner or the practitioner of the study, or otherwise be related to either or both of them, while still being independent of the study. An external expert has no financial dependency on either the commissioner or the practitioner, nor any political or other interest in the study results.

The necessary qualifications for reviewers performing a critical review depend on a number of factors, such as the type of review scheme (a posteriori or concurrent) and the goal and scope of the LCA:

  1. 1.

    Critical review practice is essential for at least one reviewer who has to be well experienced with the process of a critical review according to ISO 14044 (2006a) and ISO TS 14071 (2014). For a panel-based critical review this will usually be the chairperson of the review panel .

  2. 2.

    LCA expertise: As a general rule, there has to be at least one expert on LCA methodology and practice as well as the ISO 14040/14044 requirements. For a panel-based critical review this will usually be the chairperson of the review panel.

  3. 3.

    Technical expertise mostly concerns the LCI phase and is required in order to ensure that the underlying product system model and data are representative and adequately modelled according to goal and scope of the LCA. Technological experts do not necessarily have to be familiar with the LCA methodology. This may comprise specific expertise, such as on the

    • Product, service or organisation ,

    • Process(es) and technology,

    • Relevant practice(s) including national or regional specificities if needed.

  4. 4.

    Scientific expertise may be required to ensure adequate consideration of environmental and/or social issues and phenomena of relevance for a given goal and scope definition. This applies particularly to the LCIA phase, but may also be relevant for aspects of the other LCA phases, notably the interpretation.

  5. 5.

    Other expertise may in some cases be necessary depending on the goal and scope, e.g. legal issues, stakeholder concerns, NGOs, etc.

Proficiency in the language of the study is of course required from all reviewers. The number of reviewers or review panel members is then a function of the expertise required and the expertise each reviewer brings into the process. So far, there is no official accreditation or certification required (or available) and the expertise of reviewers will usually be evaluated via their curriculum vitae including a list of relevant references. ISO TS 14071 (2014) also proposes an example of a self-declaration statement that can be used. To the authors’ knowledge, several organisations intend to establish critical reviewer databases , but until the finalisation of this book no database has reached formal recognition in the global LCA community.

The selection of reviewer(s) will typically be done by the practitioner and/or the commissioner of an LCA. In the case of a panel review, they appoint an external independent expert as chairperson, who then selects other independent experts for the review panel. All experts are contracted by the commissioner or practitioner. This contract normally involves adequate remuneration, a commitment to tasks and timing and a non-disclosure agreement to ensure confidentiality of information and data that need to be accessed by the experts in order to complete the critical review . The reviewers’ contract cannot contain conditions that influence the result of the critical review process. ISO TS 14071 also explicitly states that reviewer tasks cannot be subcontracted or delegated and thus have to be performed by the contracted reviewers personally.

ISO TS 14071 lists the respective tasks of the two principal roles in a critical review process, the chairperson and the reviewer. The reviewer’s role essentially involves:

  • Commenting on the LCA report (or parts of it during a concurrent critical review);

  • Contributing to the critical review report;

  • Expressing agreement or disagreement concerning the critical review statement including a justification in case of disagreement.

The chairperson has the same role as a reviewer but with the additional tasks of running the critical review process via:

  • Setting up of the review panel ;

  • Distribution of tasks relative to each panel member’s competencies;

  • Coordination of the review process including ensuring a common understanding of the required tasks among all reviewers and their relation to ISO 14040 and 14044;

  • Recording and sharing each reviewer’s comments within the panel and with the practitioner/commissioner;

  • Resolve potential conflicting positions between reviewers, aiming at a consensual critical review statement or if that is not possible including a minority position in the statement;

  • Enable and support a smooth communication among all panel members and with the practitioner and commissioner ;

  • Ensure the generation and panel approval of review report and statement.

Consequently, the workload and required experience level regarding the critical review process will be higher for the chairperson which should be reflected in the contractual conditions. Furthermore, this also means that besides technical qualifications, the chairperson should be particularly skilled in communication and project management.

How to become a critical reviewer is a frequently asked question and there are several ways once an interested candidate has acquired the necessary competencies and experience. The opportunity to participate in such a process may come via different channels, typically via colleagues who may have been asked first and refer to you, via mailing lists, or via a direct contact with an offer. In any case, it is advisable to first participate as expert in a reviewer panel a few times in order to get acquainted with the process and usual practice. Having participated in a few critical reviews, you could propose to take on the additional responsibility of acting as chairperson.

4 Conclusions

Critical review is an important element of an LCA study that helps ensuring conformance to the relevant LCA standards ISO 14040 and 14044 and thus building credibility and trust in its methodology, data, results, the robustness of its conclusions , and ultimately increasing its acceptance among stakeholders . In the authors’ experience, critical review can trigger a tremendous improvement of an LCA’s rigour , transparency , technical quality and robustness, especially if conducted concurrently to the study. It also helps bringing in external and independent views and experiences, which typically enriches the methodological aspects, such as modelling, data, and the interpretation of results in a study. It is, however, not a guarantee that the study is perfect or even as perfect as possible since there will always be aspects that could not be considered, that were overlooked, or that could not be addressed. Also, a critical review does not verify or validate the goals of an LCA or how its results will be used (ISO 2006b), which means that if the objective itself is problematic the LCA study may be stated as conform to ISO, while still supporting misleading conclusions or recommendations beyond the LCA report. Critical review is also not a validation process against objective evidence, such as measurements or other observations.

This chapter provides a broad overview of several complementary aspects related to the critical review, without discussing them all in detail. The authors recommend the cited references for further reading, particularly the ISO standards of course, along with the publications by Klöpffer (1997, 2005, 2012), Weidema (1997), Hamilton and Ayer (2013), and Curran and Young (2014), which will provide further depth and details, practical aspects, and experiences for the interested reader. They are certainly essential reads for aspiring critical reviewers and chairpersons, but also for practitioners of an LCA that will be exposed to a critical review.