Keywords

1 Introduction

The essence of project management is the use of information, knowledge, skills, methods, techniques, and tools to manage some unique and complex activities to meet the previously established project requirements and provide unique products, services, or results (PMI 2017). Implementation of projects in an efficient and effective manner is important both for commercial and public projects, including those implemented in the health and medical sectors. In the case of failure of healthcare projects (related to the delivery of their products/outcomes of adequate quality in a timely manner, and possibly at relatively low costs), they have negative implications not only for management entities, but also negative social consequences.

The risk of failure is particularly large for innovative healthcare projects, which concern, for example, the construction of medical facilities equipped with highly specialized tools, the implementation of electronic circulation systems for patients’ medical records (e.g., for HIS hospitals—Hospital Information System, medical facilities EHR/EMR—Electronic Health Record/Electronic Medical Record), implementation of various types of IT projects, development of new medicines and medical devices by pharmaceutical companies (Sa Couto 2008). This risk can be reduced by using appropriate evaluation methods and systems based on properly selected indicators and criteria.

Research achievements in the discipline of management, as well as in the field of project management and evaluation mainly concern commercial projects implemented by organizations primarily focused on financial profits. Only recently research on public management as well as public project management was developed and is not yet sufficiently represented in scientific literature.

The management of public projects in comparison with managing commercial projects is fundamentally different and much more complicated (Gasik 2016). Such projects usually have multifaceted goals, and their evaluation similarly has a multifaceted character as well as unique and complex evaluation process which should take into consideration earlier planned goals, principles, sophisticated (traditional or AI-related) methods and models based on collected and analyzed information and knowledge (Grzeszczyk 2017). The product of this evaluation refers to the systematic process of determining the merit, worth, or value of public projects (Scriven 1991).

In processes of building general models of public projects evaluation and innovative projects in the healthcare sector, praxeological theory may be useful, which is universal in nature, and in addition to other applications, it may be used, for example, to build systems indicators and evaluation criteria. Particularly interesting are the general praxeological criteria of evaluation (effectiveness, economicality, and efficacy) that can be the basis for developing criteria useful in specific evaluation processes, e.g., public and healthcare projects.

The main aim of this chapter is to examine the applicability of praxeological theory to the evaluation of innovative projects in the healthcare sector. The specific objective of the study is to propose a set of project evaluation criteria and indicators. The authors undertake research aimed at filling the identified research gap in the use of praxeological theory to evaluate projects, in particular healthcare projects. The topic briefly discussed in the chapter can be successfully continued in further studies concerning larger research samples.

2 Healthcare Projects Management and Evaluation

The term “Healthcare” can be understood as maintaining and restoring health through treatment and disease prevention (Healthcare 2017). The definition that has been in force since 1948 specifies the concept of “health” as a state of complete physical, mental, and social well-being, not just the absence of disease or disability (WHO 2017).

The definition of healthcare projects was adopted as a set of projects whose essence is research, development, and innovation in the field of healthcare, medicine, both public (governmental) and private (implemented by enterprises). The clinical activities can include clinical trials whose goal is to broaden medical knowledge related to treatment, diagnosis, and diseases prevention (U.S. National Library of Medicine 2017). In addition to clinical study, the essence of which is to determine the safety and effectiveness of therapy, there are also observational studies that assess the results of treatment in groups of participants (U.S. National Library of Medicine 2017).

R&D is a systematic creative effort that aims to increase the knowledge resources used for new applications, covering the following three activities (OECD 2002):

  • Basic research, i.e., experimental work undertaken to acquire new knowledge, especially in the field of explanation of phenomena and observable facts—these activities are only theoretical.

  • Applied research, i.e., activities similar to basic research, but, in contrast, they are focused on practical goals and solutions.

  • Development works, i.e., activities based on already available knowledge (obtained, e.g., in the course of basic research), the aim of which is to create new or significantly improved products: services, products, processes, and systems.

In the literature on the subject, some examples of problems occurring in the field of R&D projects in healthcare can be found. One of the main reasons for failures in these projects, as in the case of other types of projects (e.g., commercial, production and others), are errors in communication, misunderstanding the needs of customers and services recipients (Simon and Canacari 2012).

Healthcare project management is associated with many challenges and problems to be solved. The most important external stakeholder for organizations operating in the area of healthcare are patients who should be involved in the implementation of healthcare projects, because it allows a better understanding and appreciation of the work of others in the team, and thus leads to a more efficient implementation of the goals (Simon and Canacari 2012).

In healthcare, interactions with entities such as medical schools, professional associations, or suppliers of medical equipment are important in terms of increasing the access of the project team to information and knowledge. Another challenge is the need to assess the value of external knowledge contributed to the project (especially when it concerns product innovation). The experience of the project manager is also important, which may contribute to reducing costs by discovering new knowledge in research processes, motivating the team for greater creativity, and reducing uncertainty in the commercialization processes (Salge et al. 2013). Project managers should also support the use of transparent evaluation procedures, the reliability of their results, lower subjectivism, and stimulate generation of innovations (Mikulskiene 2014).

Scientific research work, knowledge, and new technologies are an important driving force of economic growth. The strategy of R&D activities undertaken in the public sector and healthcare should refer mainly to the adopted key competences, not include activities that can be carried out by entities operating in the private sector, focus on the implementation of socially beneficial public policy, stay away from commercial interests of the private sector, support the cooperation of scientific units with enterprises, as well as the implementation of interdisciplinary and interregional research (Procca 2008).

The main goals of healthcare projects (e.g., concerning clinical trials or observational studies) should have a clear public and social character. Clinical study is usually aimed at verifying whether treatment or prevention of diseases is safe and effective, and the implementation of this type of research positively influences the progress in science; thanks to these activities, there is a real chance to help scientists find new and better treatments (NIH 2017). There is no standard defining the criteria for the success of R&D projects in medicine and healthcare. In a sense, the praxeological measures of project performance are, for example, seven success factors and rules formulated below (Duke 2015).

Firstly, the implemented or planned projects should be part of a wider, strategic plan. If projects are related to the organization’s strategy, it is possible to place them in the hierarchy of priority projects to be implemented and at the same time it is possible to plan appropriate financial resources for their implementation. Secondly, the use of benchmarking makes it easier to plan expenses, while comparative analysis of projects allows reducing the risk of unplanned costs. Thirdly, projects should not be launched until the critical preparatory activities have been completed. Accelerating work and avoiding key tasks (or shortening them, thus reducing the quality of work) causes a high risk of a significant increase in costs and extending the duration of projects. Moreover, the method of selecting the members of project teams is very important. The personnel selection process should not be conducted separately for individual tasks in projects. The integrated approach, consisting in the creation of entire teams cooperating with other teams operating in organizations, saves time connected with the organization of the working group. Another crucial success factor is that team communication should not take place between uncoordinated organizational divisions, called “siloses,” as it limits creativity and introduces bureaucratism. An efficiently implemented project should give team members the possibility of effective communication between various sections, departments, and divisions.

What is more, the entire project team should be responsible for the budget, the scope of work, and the schedule for their implementation. For the success of a project, it is important to create responsibility for the work performed as well as sharing information. Benefits for the performance of projects will be achieved by the joint work of managers and employees in the area of budget preparation and implementation of its necessary changes. Finally, the goals of projects, including their measurable effects, should be clearly communicated to all team members at every stage of work. This will translate into the quality control of the planned products and will help to protect against the closure of projects, not in the form in which it was planned.

In addition to the abovementioned success factors, the skills of the project manager are also important, as well as the appropriate selection of an interdisciplinary team that copes well with the use of multifaceted evaluation criteria and indicators (Payne et al. 2011).

3 Evaluation Criteria and Indicators

In the case of healthcare projects, the evaluation problem is multifaceted; it is not possible to limit evaluation processes to only one criterion (e.g., financial efficiency) and a multi-criteria evaluation should be used. Evaluators should also pay attention to designs of multidimensional approaches that combine qualitative and quantitative benefits as well as financial and nonfinancial means of projects, programs, project portfolio, and levels of enterprises (Glodzinski 2018).

The evaluation criteria make it possible to organize systematic research that supports the improvement, development, acquisition of knowledge, and better understanding of evaluated individual activities, projects, and project collections (portfolios and programs) (Rossi et al. 2004). After defining the evaluation criteria, the data collection plan, set of objectively verifiable indicators (supporting monitoring of progress in projects implementation), and sources of their verification must be also specified.

There is no single standard for setting out the evaluation criteria for R&D projects. The criteria may be, for example, the organizational value (evaluated are aspects such as the impact of projects outcomes on the organization’s ability to cooperate in the future), social value (understood as public goods, social value of projects for the city, region, country), professional value (main objectives achieved, acquired competences of researchers and the entire organization), economic value—considered as the creation of new jobs, economic growth, development of innovation (Mikulskiene 2014; Kumar 2004).

In the case of innovative projects evaluation, the following criteria can be applied (Lis 2012):

  • The real cost of the project compared to the expected cost

  • Time to market the products of the project

  • Profitability of the project and return on investment

  • Value for shareholders

  • The level of technological advancement of the product

  • The degree of meeting customer requirements (customer satisfaction)

  • Number of registered patents and the value from the sale of licenses

  • Product quality.

The individual criteria are assigned an appropriate system of measures and weights, which aims at increasing transparency, limiting subjectivism, as well as facilitating the identification of innovative ideas. Criteria can also be divided into certain categories, for example: financial, economic, organizational, technical, strategic and social. Table 1 shows further examples of evaluation criteria for R&D projects divided into three groups.

Table 1 Groups of R&D project evaluation criteria
Full size table

The European Commission is coordinating the Horizon 2020 program—the largest EU program focused on research and innovation (European Commission 2017b, c). Under this program there is a competition for co-financing activities in the area of “Health, Demographic Change and Wellbeing,” which supports activities that increase the quality of life of older people as well as the development of new, safer, and more effective treatment methods (European Commission 2017d). The following evaluation criteria for projects implemented under Horizon 2020, in some sense, are in line with the praxeological approach (European Commission 2017a):

  • Excellence—understood as clarity and accuracy of goals, solidity of the concept, credibility of the work methodology, the degree of exceeding the current state of the art (groundbreaking goals, new concepts and products), taking into account the interdisciplinary approach, using the knowledge of stakeholders

  • Impact—the degree of implementation of the expected effects of projects, increasing the innovative potential, creating new market opportunities, strengthening the competitiveness of enterprises, creating benefits for society

  • Quality and efficiency of the implementation—understood as the quality and effectiveness of the planned work, the appropriateness of management structures and procedures (risk, innovations), the degree to which knowledge is combined (in the case of a consortium), the correctness of the distribution of tasks and the possession of appropriate resources.

In the above case, the evaluation criteria are ex ante, as they are applied before the projects implementation, at the stage of submitting applications for co-financing.

INNOMED and InnoNeuroPharm are two examples of programs managed by a Polish institution called National Centre for Research and Development (NCRD, in Polish: Narodowe Centrum Badań i Rozwoju) supervised by Ministry of Science and Higher Education. Both programs are part of the Smart Growth Operational Programme (SGOP) 2014–2020, and their aim is to work in the field of medical engineering technology (including medical biotechnologies in oncology), diagnostics and therapy (including personalized therapy in cancer treatment), manufacturing medicinal products for applications in oncology (NCRD 2017a, b, c) as well as in the field of innovative medicinal products, innovative technologies for the production of medicinal products, innovative diagnostic methods, innovative methods of rehabilitation, development of tools supporting research into medicinal products (NCRD 2017d).

The set of indicators used to evaluate these programs is presented in Table 2. The evaluation is carried out during the implementation of projects and after their completion. In addition, these indicators are used to study project impact and life science innovation, which is up to 5 years.

Table 2 Indicators used to monitor and evaluate selected healthcare programs
Full size table

The indicators used to evaluate the INNOMED and InnoNeuroPharm programs are similar since both are included in one operational program—SGOP. In addition, among the indicators supporting the monitoring and evaluation of projects co-financed from EU funds are: the number of enterprises supported in the scope of conducting R&D work, increase in employment in supported enterprises.

One of the interesting tools used by organizations operating in the healthcare industry is ADLI (Approach, Deployment, Learning, Integration). Approach and Deployment elements are present in the active phase of the project life cycle, while Learning and Integration relate to the final phase of the project. All these elements, used as a checklist at the stage of project summary, help in evaluating the results of the project. Murphree, Vath, and Daigle propose the use of a checklist (Table 3) before the project finishes and goes to the final evaluation—thanks to this, problems with the project’s organizational efficiency can be solved easily and quickly. The abovementioned tool is used as part of the Lean Six Sigma approach (Murphree et al. 2011).

Table 3 The essence of ADLI tool
Full size table

Previous studies on criteria and indicators of healthcare projects monitoring and evaluation can be considered insufficient. There is a possibility to apply praxeological theory in processes of assessment of healthcare management performance, which allows for taking into account difficult-to-measure features (e.g., people’s attitudes and behaviors, conditions shaped by qualitative factors), and three determinants of praxeological performance: management conditions, managers as a doer, and goals of management (Striker 2014). Human factors are so clearly marked, because praxeology regards activities as a human behavior that is voluntary, conscious, and oriented toward some purposes (Gasparski 1996).

It is worth undertaking a pilot study in this area based on praxeological theory and criteria of effectiveness, economicality, and efficacy for evaluating projects performance. Effectiveness is connected with achieving desired results of projects. Praxeological economicality is a concept similar to efficiency and makes it possible to compare the achieved project outcomes with the input resources. In classical praxeology, efficacy is determined as the difference between acquisitions (favorable project outcomes) and losses (resources used and wastage) (Pszczolowski 1978). Criteria of this kind should take into account not only quantitative aspects, but also difficult-to-measure qualitative factors.

4 Pilot Study and Results

As part of empirical research, a pilot study was conducted in the form of a mini focus group interview (Mini FGI) among the management of one of the Contract Research Organizations (CRO) operating in Poland. Interviews with experts were aimed at identifying key indicators and evaluation criteria of performance of healthcare projects.

The research used a previously developed list of criteria and indicators for R&D projects, without taking into account the specifics of individual fields (Grzeszczyk and Zawada 2017). As a result of this study, the usefulness of these criteria was confirmed in the healthcare projects evaluation. The research included the following healthcare projects: R&D projects, clinical/observational studies and other public projects in the field of health and medicine.

Experts are the management staff of healthcare projects in one of the CRO. They have over 5 years of experience in managing R&D projects, which are implemented jointly with Polish public scientific units (universities), as well as commercial projects in the medicine and pharmacy industry. Each of the interviewees took part in at least five healthcare projects, including large projects in the field of multicenter clinical research. Positions held by individual persons are: President of the Board, Member of the Board, Director of Operations, Medical Director, Business Development Manager. The meeting of the focus group took place in December 2017 at the headquarters of the CRO.

Focused group interview was carried out in several stages. The first of them introduced participants to the subject of the research and presented the research problem. The definition of particular terms in the field of praxeology was clarified. Subsequently, the research moderator asked all participants to address the criteria and indicators for evaluating the performance of the projects. This part of the conversation was divided into three sections concerning, respectively, the following evaluation criteria: effectiveness, praxeological economicality, and efficacy.

Experts were asked about the extent to which they agree with the statement that the indicators and criteria listed in the table are adequate, important, and useful in practice for the healthcare projects evaluation. The study used a 7-point Likert scale. The result of the Mini FGI and key criteria and indicators are shown in Table 4.

Table 4 Criteria and indicators of the praxeological evaluation of healthcare projects performance
Full size table

Some of the more important conclusions resulting from interviews with experts are as follows:

  1. 1.

    In the case of research projects, it is often difficult to plan them accurately. For this reason, discrepancies between the assumed schedule and the actual duration of the project should be accepted.

  2. 2.

    The correct monitoring of the research project is of key importance. Delay times should be monitored for individual project phases, but without a precise reference to the overall action plan.

  3. 3.

    In the correct implementation of projects, the experience of projects teams plays a very important role. It should not only have a quantitative nature (measured by the number of previously implemented projects and their values), but also have a nonmeasurable, qualitative character, referring, e.g., to the characteristics of projects managers. They should show a lot of commitment, identify with the accepted goals of projects, and accept them mentally.

  4. 4.

    Innovation is not so important. Many projects in the field of medicine do not consist in the development of new drugs, but in the development of, for example, a new method of treatment, i.e., a new application of existing substances.

  5. 5.

    The development of new utility models and patents is not a very important element either. The patent application requires the publication of quite detailed information about the product in the patent registry after its production and full implementation into business. The patent application enables other interested entities to launch a product with similar features on the market, and as a result may adversely affect the competitive position of the company.

Taking into account the analysis of literature and conducted research, the following evaluation criteria for healthcare projects were formulated:

  1. 1.

    The set of stakeholders

    Stakeholders are internal and external units involved in the project implementation. The internal stakeholder can include a project team, other employees of the organization, or top management. External stakeholders include customers, suppliers, competitors, business partners as well as social organizations, government organizations and many others. A wide range of project stakeholders, and thus planned project products and outcomes, can have a beneficial effect on the quality of work results. Taking into account the requirements, expectations and suggestions possible for many environments will have a positive impact on the value of the work results. As a measure of the scope of the set of stakeholders, the authors propose the number and diversity of entities.

  2. 2.

    Quality of external knowledge

    As the existing research results shows (Salge et al. 2013), greater efficiency is characterized by projects that are implemented in accordance with the exploratory approach, i.e., regarding the search for knowledge outside the organization, including in accordance with the philosophy of open innovation. The use of external knowledge and technologies, in contrast to the operational approach, promotes the increase of creativity and accelerates the development of innovative projects products. To assess external knowledge, one can apply the following: analysis of the possibility of using new technologies in other fields of exploitation, scope of patent protection (if applicable), scientific experience (academic titles), and publishing achievement (in the case of human resources).

  3. 3.

    Transfer of knowledge

    The flow of knowledge and information in the project is in most cases a closed circuit, runs within the organization, between the members of the working group. From the point of view of the efficiency of project implementation, especially of a research and development nature, the correct knowledge management process is important, including its creation, processing (using), and storage. To assess knowledge transfer, it is possible to analyze the knowledge transfer methods present in the project organization, such as a network of links, documentation, instructions, training or databases (knowledge bases). In addition, the assessment of the willingness to share knowledge among team members may help to determine the tendency to creatively search for solutions.

  4. 4.

    Leader’s experience

    The manager, who should have rich professional and scientific experience, is responsible for the success of the innovative project implementation. The manager’s knowledge and the ability to communicate with all team members have a significant impact on the smooth implementation of the project. Effectiveness of the manager’s actions manifests itself in the ability to overcome problems encountered in research and development processes. Among the measures of experiences, one can indicate the period of professional experience in a managerial position, experience in work at various stages of R&D processes (basic research, industrial research, experimental development work, implementation phase), number of patents obtained or value of completed projects.

  5. 5.

    Communication

    One of the most important elements of the proper functioning of the project team is proper communication. Good organization of work achieved through clear communication of tasks, freedom of discussion and expressing own opinions, taking into account the opinion of all team members and providing feedback to each other, contributes to reducing misunderstandings, errors and delays in the implementation of research. The aspect of communication quality can also be considered by the number and frequency of meetings of the project group.

  6. 6.

    Teamwork

    Proper distribution of tasks can significantly contribute to increasing the efficiency of work. When activities are carried out in a collective way, it is easier to achieve the intended effects in terms of budget, quality, and time of project implementation. The quality of team work can be measured by answering the following question: How many processes and activities in the project are performed by a team of employees and what part of the work is entrusted to individual employees? In the case when the effect of the project is to be a new product, the synergy effect is possible with the use of an isomorphic project team structure (Miterev et al. 2017).

  7. 7.

    Team interdisciplinarity

    The diversity of knowledge and experience of team members can significantly contribute to the improvement of work efficiency, and thus to better project outcomes. An efficient and effective team of people has a much better chance of achieving success when the team members include not only technical staff (researchers), but also professionals in law, finance, management as well as technology developers, bioinformaticians, statisticians, and many others.

  8. 8.

    Compliance with the plan

    Material and financial schedule and appropriate classification of activities in relation to individual project phases (research phase, development phase, implementation phase) are other key elements of project management of an innovative nature. An important aspect is also proper planning and maintaining compliance with TRL—Technology Readiness Levels. Due to the peculiar character of R&D projects, it is difficult to accurately predict the results of works. Therefore, efficient schedule management significantly influences the success of the project. To assess the degree of implementation of the plan, it is possible to use a measure of time deviation from the assumed dates of the implementation of successive milestones of the project.

5 Conclusions

The publication briefly presents the contemporary problems of management and evaluation of innovative healthcare projects, which are relevant to the development of the theory and practice of management sciences. The conclusions drawn from the research results make it possible to formulate the statement that it is possible to use the praxeological theory in the evaluation of innovative healthcare projects. The more important results of the conducted research and analyses are formulated key evaluation criteria and indicators for assessing the performance of this type of projects, based on effectiveness, praxeological economicality, and efficacy.

The terminology used in the publication is not unequivocal and equally interpreted by different researchers. The literature of the subject is available in various languages, which makes it more difficult to use uniform concepts. For a better understanding of the ideas referred to in the chapter and further development of research in this area, it is necessary to look for the meaning of the concepts used in certain contexts and applications. In practice, praxeological criteria are rarely implemented in direct form and this is also the case for the innovative healthcare project evaluation.

The proposed criteria and indicators may be a complement to the ones previously used. These problems may be further explored using a larger research sample, which may provide greater credibility to the conclusions. It is also possible to consider expanding the scope of research by experts (research workers in scientific units and universities) as well as worth considering the usefulness in the evaluation processes of various scientific trends of praxeology.