Introduction

Progress resulting from research activities has contributed to improvement of the quality of life in many different domains, such as the sciences, health, and economics. However, the repercussions of research activities are no longer unanimously regarded as positive, due notably to misuse, misinterpretation, and fraud. In the field of sanitary safety, research results are of prime importance as they serve as the basis for public decisions either directly or indirectly, through the work of the experts; it is, therefore, crucial that research work be of “excellent quality”. In order to reassure the users (the scientific community, the entities using the research results for development purposes) and stakeholders (society) of the research results, about the reliability and the credibility of these results, reflection was initiated worldwide about the implementation of a quality process in the entities involved in academic research or research and development (R&D) [1–9]. Considering the experience and the knowledge capitalised in the field of QA regarding analytical activities, the first question was to determine whether the existing standards used in the analytical laboratories (ISO 17025 [10]) and used as part of experimental studies performed in pharmacology and toxicology, for instance (Good Laboratory Practice (GLP) [11]) could be satisfactorily applied to research activities. Although the use of a single QA system has been proposed for both routine and research activities [12], it is now more and more accepted by the scientific community that specific standards are required or preferred when performing research activities. This opinion is based upon the limitations of the standards dedicated to analytical laboratories; these limitations arise from the nature of research activities (e.g. exploration of unknown facts, flexible approach required⋯) and from the requirements of these standards (e.g. rigid organisational structure, pre-defined methods⋯) [2, 7, 9].

The main controversial issue concerning the implementation of a QA system for research activities can be summarised by the following question: “quality in research” or “quality of the research”? Scientists who are not familiar with either the notion of quality or existing standards tend to merge both expressions. The difference between the expressions “quality in research” and “quality of the research” reflects the confusion that arises from the word “quality”. Indeed, in general terms quality implies “a level of goodness or excellence that provides satisfaction” [9], whereas in the context of QA it has a totally different meaning, as it suggests compliance with requirements imposed by a specific quality standard which is intended to give satisfaction to clients. In other words, the “quality in research” approach focuses on the way the research activity is conducted, in terms of quality requirements. However, the expression “quality of the research” refers to the excellence of the work (in terms of results and progress of knowledge), which is usually evaluated by peers through different processes such as the publication of an article in a scientific journal. To summarise, peer evaluation focuses on research results, their interpretation, and the way the research activity has been performed from a scientific and a technical point of view, whereas in quality in research the emphasis is put solely on the conduct of the research, and evaluation of the results falls within the competence of peers.

The different concepts of what a QA system in research should be reflect the uneven progress in the reflection initiated in different countries and entities, and resulted in the proposal of several standards at a national [4, 13–18], a European [19, 20] and an international level, with the ISO 10006 [21] which is one of the most promising standards for the entities performing research activities as projects.

In France, the reflection was initiated on the request of the research ministry and sustained by the French Standardization Association (AFNOR), as part of a commission and working groups created for that purpose and which gather prestigious colleges, universities, and many other entities performing research activities in various fields (health, food, agronomy, geology, energy, transport, telecommunications, etc.). The French Food Safety Agency (Afssa) is one of these entities and our participation in the work of the AFNOR was beneficial for the “design” and the implementation of a QA system for research activities. Reciprocally, the AFNOR derived some benefit from our progress in the field of quality in research.

In this paper, we present the QA system being experimentally implemented in the Food and Food Process Quality Research Laboratory of the Afssa, to carry out research activities related to contamination of the food chain by toxins and other contaminants, in compliance with the quality requirements we identified. Furthermore, the specificities and originalities of this system in comparison to the other standards are depicted.

Finality and typology of research activities at the Afssa

The Afssa is a public body under the supervision of the ministers in charge of health, agriculture, and consumption. It was created in July 1998 by the law related to Health surveillance. The mission of the Agency is to provide and ensure the sanitary safety of food intended for the human consumption, from the production of raw materials to distribution to the end-users. The Agency also has specific missions in the field of animal health and veterinary drugs. To fulfil these missions, research activities can be carried out in any of the 12 laboratories of the Afssa, as a support to:

  • the activities performed as part of the scientific and technical support provided notably to the supervising ministries; and

  • the work of the experts assigned to risk assessment in relation to specific topics identified by the supervising ministries, the recognised consumer societies or the Agency itself.

In other words, research activities performed at the Agency can be considered to support expertise and public decisions. Indeed, the experts are some of the “clients” of the research activities, as they deliver their opinions notably on the basis of the conclusions of the research projects; furthermore, these research activities are intended to directly or indirectly (through the expertise conclusions) serve as the basis for decisions taken by the supervising ministries in relation to the missions of the Agency. As these exploratory activities concern risky domains (sanitary and economical), they must comply with specific quality requirements to prove the reliability and credibility of the results and to keep traceability of the experimental studies through the years. Moreover, the scientific community is another “client” of these research activities, very demanding on the excellence of the results, both at a scientific and a technical level.

At the Afssa the research activities mostly belong to the field of applied research rather than fundamental research and are conducted as projects, defined as a global process composed of several coordinated and controlled activities, which are conducted to reach an objective in compliance with specific requirements, including constraints on time limit, costs, and resources [22]. Perennial activities such as thematic actions are also performed at the Afssa in order to maintain and develop the competence of the units, and thereby enable exploratory and innovating actions to be undertaken when the knowledge already capitalised is not sufficient to conduct a project [16].

Quality requirements for research activities as perceived by the reflection committee

A “reflection committee” incorporating the quality Delegate of the Afssa was created in the Food and Food Process Quality Research Laboratory to design and implement a quality-assurance system in research in this laboratory. This committee identified several requirements that had to be met, to give confidence in the research results.

Total quality management

In the field of QA, a means of proving the reliability and credibility of the research results is to demonstrate that critical points in terms of quality requirements are controlled throughout the whole research process, from the very beginning until the end. This global approach, which enables the total quality management (TQM) of research activities, requires the active participation of all the actors: the scientific and technical staff, and all the people associated with the support activities required to perform research activities (the administration and financial service, the legal service, etc.). The way to achieve the TQM is to implement a quality-management system (QMS) and to comply with the corresponding requirements. The architecture of the documentation associated with the QMS can vary from one research entity to another, but it should at least include a quality manual, describing clearly the quality requirements to be met. The second documentation level we identified is the quality plan, also known as the research plan [17, 18, 20], which is the written format of the research project. Every project should be associated with a quality plan defining the objectives to be met, the tasks involved, the constitution of the research team (including the appointment of the project leader), the associated resources, the milestones, the time limit imposed or planned, etc. Research activities performed as thematic actions equally have to be accompanied by a quality plan. The third documentary level, generally common to all research activities (projects, thematic actions) is constituted by the general and specific procedures.

Technical competence and quality management

These are the quality requirements that constitute the foundations of the analytical activities and are evoked in the corresponding standards, such as ISO 17025. However, these requirements are not exclusively intended for routine analyses but also partly apply to research activities. Indeed, the teams performing research activities in our laboratory also carry out routine analyses in compliance to the requirements of the ISO 17025 and, therefore, are well aware of the quality requirements regarding the personnel, the materials, and the premises, relevant for both types of activity (routine and research). A research laboratory which is not familiar with quality assurance should first take its inspiration from ISO 17025 for the requirements related to technical competence (TC) and quality management (QM). As the quality requirements applicable to research activities and identified by the reflection committee are not restricted to TC and QM, ISO 17025 is not sufficient on its own and must be completed by other standards or other documents.

Project management

On the basis of the decision made by the General Management of the Agency, research activities are mainly performed as projects. This implies some specific requirements due to the nature of the research and in relationship with the defined objective, the explicit beginning and ending, the actions to be undertaken, and the limited allocated resources. For instance, it is crucial to monitor and to evaluate the project from the beginning until the end, to make sure that the time limits, the costs, the resources, and the risks associated with the project are planned and controlled at any time. For research activities performed as thematic actions, the requirements of project management can be adapted in order to be more flexible and suitable for thematic actions.

Control of quality and the availability of resources

At the risk of being redundant because these aspects are part of both the project management (PM) and the quality management (QM), we would like to focus on the requirements arising from the strict control of the resources as it is a key issue. Furthermore, in research activities, scientific competence (SC) is of great importance as it constitutes the foundation of the exploratory process and needs to be optimised. A means of achieving this goal is to promote a working atmosphere beneficial to research (participation to conferences, scientists welcome, scientific watch⋯). The notion of scientific watch is very important for a research activity, as it constitutes the starting point of the entire activity and is a tool essential for SC. Indeed, when a need for knowledge emerges, the first step is to determine whether an answer can be found in the literature. A research project is initiated as soon as the question remains unanswered, then this unanswered question is transformed into a query raised; this constitutes the “problem-building phase” which is followed by the formulation of hypotheses. Furthermore, the watch process has to be carried out throughout the whole research process to inquire about the findings in a specific domain and eventually for reorientation of the research axis.

Specificity and originality of research activities

Besides the requirements mentioned above and already taken into account in some standards and guidelines dealing with quality in research, we identified a number of quality notions specific to research activities that are original, in the sense that they have never been identified or reported previously. These notions will be discussed later; they are of great importance to research activities.

Pros and cons of existing quality standards

On the basis of the quality requirements identified above and applicable to research activities, the reflection committee searched for a quality standard that was able to fulfil its expectations, in terms of QM, TC (both necessary to carry out the routine analyses), PM, and SC.

Concerning the QMS, the entities involved in the design, development, and production of goods and services are familiar with standard ISO 9001: 2000 [23], which presents the requirements of a QMS. However, these provisions are not satisfactory for research activities, because the conformity of the final product is strictly defined as part of the requirements imposed by the client and this constitutes a major requirement of the ISO 9001: 2000, but in research the results do not necessarily match the initial expectations, can be unknown or even not conceivable, without affecting the validity and the relevance of these results; this is a key issue that will be discussed later. A commission has been created under the aegis of the AFNOR to supervise the work aiming at adapting ISO 9001 for research entities.

The TC and the QM requirements relevant for research activities are those described by standard ISO 17025, applied in our laboratory when performing routine analyses. However, some of these requirements are either incomplete or too restrictive to apply, in that state, to research activities. This is the case, for instance, of the non-conforming work, which has a negative connotation for a calibration and testing laboratory but can be promising for a research entity (this point will be discussed later). However, concerning the TC, it is hardly conceivable that every method be validated in a research project, according to the same requirements as those of routine work. Indeed, this would be far too constraining, provided that a method may only be used for a specific part of the research work. Furthermore, ISO 17025 does not describe how to record and organise data regarding the conception of experiments (data presented in a quality plan) and how to implement a working atmosphere favourable to the research.

Regarding the guidelines for quality management in projects, standard ISO 10006 [21] gives some indications of the way to run such a QMS and therefore, it is relevant for the QMS in research to be implemented at the Afssa. This standard is recommended for those who perform research activities as projects and are not familiar with the notion of quality management. However, this standard does not incorporate notions of prime importance to research activities such as the distinction between positive and negative non-conformity or even the notion of non-confirmation of hypotheses, presented below.

The standards discussed above do comply with requirements regarding the QM, the TC, the SC, or the TM but none of them covers the requirements fully. As research activities should incorporate all these requirements, we studied other specific standards developed for that purpose, in order to determine if they could fulfil our quality requirements.

Thus, NEN 3417 [18] and the standard of the BELTEST [17] have been proposed in the Netherlands and in Belgium, respectively, as an amendment of EN 45001 [24], now replaced with ISO 17025. These standards incorporate the QM and TC requirements of the EN 45001 and specify that the research activities should be performed as projects, although there is no explicit requirement concerning PM aspects. Both standards incorporate the notion of a research plan, evoked earlier. Furthermore, the Belgian standard is dedicated to the entities willing to go through the accreditation process for their research activities.

As for the guidelines of the European Association of Research and Technology Organisations (EARTO) [20], they are more complete than the Dutch and the Belgian standards, as they incorporate the QM and TC requirements of ISO 17025 and requirements specific to research activities run as projects. Furthermore, these guidelines also incorporate the notion of a steering committee to control and direct the execution of the project, which is an important aspect as part of the SC. The pilot guide for quality in research [4] also describes relevant requirements that apply to the different stages of a research activity.

Considering our quality requirements in terms of QM, TC, PM and SC, we implemented a quality-management system in research (QMSR) based upon the quality requirements shared by different standards and not necessarily specific to the research activities: that is, the QM and TC requirements of ISO 17025 also applicable to research activities and the PM provisions inspired by ISO 10006 and the EARTO. The emphasis is also put on the resources (human, material, and financial) controlled by the project leader and the steering committee.

Furthermore, to fulfil our requirements in terms of quality in research and to take into account the specificities of research activities, we incorporated some quality requirements which have not yet been identified by any of the existing standards; these specific requirements characterise our standard from the point of view of its originality.

Specificities of the QMSR developed by the reflection committee

The notions specific to research activities that we identified concern non-conformity which can be either positive or negative, the non-confirmation of hypotheses, and the role of peers in the evaluation process, which has been reinforced.

Positive and negative non-conformity

According to ISO 17025, a non-conformity is defined as a difference with regard to the explicit and implicit specifications of the quality manual. A non-conformity can concern either a written recommendation of the quality manual (e.g. weekly control of automatic pipettes has not been done) or the application of these recommendations (e.g. as part of the weekly control of the pipettes one was found to be out of the range of the acceptance criteria). Whatever its nature (recommendation or application), in routine analysis a non-conformity has de facto negative acceptability, but in research the situation is more subtle as a non-conformity can be either negative or positive. Indeed, the typical example of a positive non-conformity is the discovery of penicillin by Dr Alexander Fleming in September 1928. While growing Staphylococci on Petri dishes, he found mouldy colonies of Penicillium notatum that contaminated his cultures. At this point, it was a negative non-conformity, but instead of discarding the contaminated plates, Alexander Fleming noticed that the staphylococci were unable to grow around the P. notatum colonies and discovered an antibiotic he named penicillin. This application non-conformity, resulting from a technical problem, illustrates the importance of the analysis of every research result, as some benefit can be derived from apparently negative results.

Non-confirmation of hypothesis

This notion, specific to research activities and every intellectual process aiming at resolving problems, can be defined as the difference observed between the hypothesis made after the problem-building phase, as part of the research process, and the capitalised results which do not confirm the initial hypothesis. A non-confirmation questions a part of or even the entire hypothesis and in that respect it can be either partial or total. Furthermore, a non-confirmation in a specific context can be beneficial in another context by providing answers to questions raised or by enabling development which had not been considered before. The Post-it note is a typical example of a discovery that resulted from a non-confirmation; indeed, while Spencer Silver hypothesised that he could improve the acrylate adhesive used in many of the 3M’s tapes, he discovered an adhesive that would not stick very strongly. The unexpected sticking properties of this adhesive have been utilised for the Post-it notes.

The notion of non-confirmation is of great importance for research activities, as the final results may not match the initial expectations but still be valid and relevant. This is the case for the unexpected discoveries, but a non-confirmation is not necessarily associated with the idea of a discovery. Indeed, it can lead to the reorientation or (temporary or definitive) abandonment of a research axis provided that the initial hypothesis on which the whole research activity has been based is not being confirmed. Due to the major consequences on the research process, every non-confirmation should be dealt with rapidly but with a special care, under the responsibility of the project leader and the steering committee. Furthermore, it is important to record and to date all the research data even the hypotheses which were partially or totally, temporarily, or definitively abandoned, in the event of a patent or to prove the paternity of a discovery.

Evaluation conducted by peers

Research activities performed as part of a QA system should be subjected to two different types of evaluation. The first concerns assessment of the QA system to determine whether the requirements applicable to the research activities have been fulfilled. In calibration and testing laboratories, this is done by a qualitician as part of an audit. The second type of evaluation is performed by the steering committee in the course of the research activity and by peers for final evaluation, carried out to assess the excellence of the research work. We propose that both types of evaluation be performed by peers familiar with the way to run research activity, in order to avoid the separation of the results and the quality requirements attached to the research activity. The steering committee would still play a role in the “in-course” evaluation. Obviously, this would require that peers be taught the way to correctly perform a quality assessment and they should be familiar with the implementation of a quality system in research. We consider that “quality is built in research” and not out of research, therefore quality assessment should be performed concomitantly with evaluation of the excellence of the research work. Universities could contribute substantially to this process by implementing a “quality in research work” module in their educational program, to increase the awareness of junior scientists of quality aspects in relation to research activities. This also implies that the senior scientists in charge of supervision of a research activity as part of a PhD must be aware of these quality requirements. Therefore, the university diploma required in France to supervise any junior scientist (PhD student) needs to be updated to take these quality requirements into account, and some universities are considering this possibility.

Conclusion

This QMSR has been developed to match as closely as possible our research typology and to fulfil our expectations in terms of quality requirements, to give confidence (reliability and credibility) in the results of research activities carried out in our laboratory. This QA system is a hybrid incorporating the requirements of several standards, dedicated to both routine and research activities, because we could not find a single standard able to comply with all our quality requirements. Furthermore, we identified some notions of great importance to research activities, which have never been reported and which the scientists should be aware of, as they clearly differentiate research activities from routine work.

The implementation of a QA system in research is a process that must take into account the specificities of the entity (in terms of research typology, quality requirements, etc.). Moreover, some benefit can be derived from the quality experience capitalised in the field of routine activities and several standards are available as a starting point. However, it is essential to consider the specificities of the research activities like those we identified. Furthermore, the notion of non-conformity has to be revised in relation to a research activity and it is necessary to go thoroughly into the notion of non-confirmation.

Considering the level of requirement of this QMSR, the provisions of the EARTO and BELTEST standards are met. This does not imply that our QA system is too constraining. Indeed, a quality system must find the right balance to be demanding enough in terms of quality requirements without being too constraining; otherwise this would be detrimental to the freedom and the creativity of the researchers and thereby restrain the whole research process.

The final product of the research has to be evaluated and subjected to criticism by peers or by referees, in the event of the submission of an article to a scientific journal. We consider that peers and referees should assess the quality concomitantly with evaluation of the excellence of the work, as “quality is built in research”.

The implementation of a QMSR will contribute to the increase in the degree of confidence in the results by removing potential uncertainties concerning the conduct of a research activity. Furthermore, it is important to give confidence in the results, in order in return to be able to trust the published results, as it is crucial for research work to be based on reliable data.

The applicability of this QMSR will be verified as part of PhD projects carried out in our laboratory. Furthermore, it will be necessary to determine whether our QMSR is compatible with the research activities performed in the other laboratories of the Afssa, eventually to arrive at a single system.