Keywords

1 Introduction

Software is successfully applied in a wide variety of areas. It supports and facilitates the activities of individuals and organizations in their daily routines. Modern world is highly dependent on software. However, software projects have been suffering from poor reputation [13, 17, 22, 23] by repeatedly bursting deadlines, costs or failing to fully meet user requirements. There are reports of software flaws reaching billions of dollars, as well as failures that have caused accidents and damage to human life [7, 8, 27].

An approach to understanding the causes of software project failures is to conduct postmortem analysis [3, 6, 14, 21]. This analysis is a collective learning activity which can be organised for projects either when they end a phase or are terminated [10, 14]. The main motivation is to reflect on what happened in the project to improve future practices for the individuals and teams that participated in the project, as well as the organization as a whole. Other terms may also represent this type of processes, such as project retrospective, lessons learned, and postproject review.

Since the 1990s, this method of analysis has already been recognized as a practice that guarantees competitive advantage to companies [18]. Birk et al. [2] mentioned that the postmortem analysis is an excellent method for knowledge management by capturing experiences and improvement suggestions from completed projects and works even in small and medium-sized companies that do not have large budgets. Several researches have been developed in order to improve this form of analysis in software projects [1, 5, 9, 10]. Even so, the answers to understanding why software projects fail are often neglected in organizations [1, 11, 12, 14], and so are not readily available.

Failure to perform postmortem analysis makes it difficult to identify indicators for improving organizations. For example, the knowledge acquired in this analysis allows modifying and improving the software development process [16], as well as identifying critical points before and during project execution [1, 14]. Sommerville [24] reports that it is possible to improve the software process because many organizations may include outdated techniques or do not take advantage of the best software engineering practices in industry. Therefore, applying techniques that aid in detection of failure points is important for prioritizing improvement actions and increasing the effectiveness of software construction activities.

Knowledge Management is a large interdisciplinary field that provides methods that simplify the process of sharing, distributing, creating, capturing and understanding of a company’s knowledge [4], allowing the organization to modify its behavior in order to reflect new knowledge and ideas.

Thus, with the premise that postmortem analysis is an important tool for improvement of realization of software projects and is often neglected, according to reports found in the literature, this work aims to identify and list the main technical and managerial difficulties for its accomplishment. For this, a bibliographic research was carried out in several scientific databases in the area of Computer Science and Management.

The reminder of this paper is structured as follows. Section 2 presents the background and general theories on postmortem analysis and knowledge management. Section 3 describes the research method. Sections 4 and 5 are respectively related to the technical and managerial difficulties in the postmortem analysis, and Sect. 6 summarizes findings and suggests future work on the topic.

2 Background

This section briefly describes the concepts of postmortem analysis and presents the phases that define its process, then it is explained how knowledge is transmitted in an organization through the theory of Knowledge Management.

2.1 Postmortem Analysis

Postmortem analysis is a collective learning activity which main motivation is to reflect on what happened in the project [10]. The objective is to identify the success and failure points of previous projects to acquire knowledge that will allow improvement in execution of future projects. This analysis can be performed when a phase ends or when a project is terminated. Lessons learned make it possible to improve the individuals and teams that participated in the project, as well as the organization as a whole [10, 14].

Postmortem Analysis is a relevant tool for project teams to collectively identify communication gaps and practices to improve future projects [3]. Postmortem analysis in software projects provides an excellent method for knowledge management, due to the high feasibility for continuous improvement and corrective actions development [2, 14].

There are variations in relation to the objective and degree of formality in the execution of a postmortem analysis. It can be focused on collecting experiences related to a simple activity, the phase of a process, or acquiring the available experiences of a project as a whole [25]. Data collection can be performed through semi-structured interviews, informally, or through a multi-step process using formal methods [25].

For a postmortem analysis a well-defined process is required [9] which can generally be simplified into four phases [1]:

  1. 1.

    Data collection - Data are collected from team members through interviews and questionnaires, or a combination of the two. Project documentation can also be used as a source for data collection;

  2. 2.

    Workshop meeting - With some members who participated in the current project, a meeting is held using formal analysis methods, such as structured discussions, root cause analysis and fishbone diagrams, to elicit tacit knowledge from participants [1];

  3. 3.

    Data analysis - In this phase the lists with positive and negative points of project are analyzed, creating an order of impact in the project, statistical methods can be used to aid in the process. This phase can be performed during a workshop meeting or separately;

  4. 4.

    Present results - Results are presented to members of the organization in order to allow others to use lessons learned in the development of future projects.

The common phases of the processes heavily depend on participation of the project team members and the subjective opinions expressed by them [1]. The role of documentation is not so important when compared to the role of workshops and interviews. Thus, most studies on postmortem analysis of projects have used methods that require active participation of project staff or their combination using project documentation.

Even though it is an important activity for improving project development, postmortem analysis in software projects is often neglected [1, 11, 12, 14], which highlights the need to disseminate such practice. However, for postmortem analysis to be more commonly performed in organizations, factors that make it impossible or difficult to execute must be found and effective measures must be taken.

2.2 Knowledge Management

Knowledge Management can be defined as a method that simplifies the process of sharing, distributing, capturing and understanding a company’s knowledge [4]. The purpose of these efforts is to provide the employees of the organization with the knowledge that they need to maximize their effectiveness, thereby expanding the capacity of the organization [19].

In the Knowledge Management, the word “knowledge” is usually classified into tacit and explicit knowledge [10]. These two forms of knowledge make up the epistemological dimension in the creation of organizational knowledge [20]. Tacit knowledge comes from interactions of the individuals that constitute the organization, through exchange of experiences, ideas, emotions and conversations. Explicit knowledge is what can be represented in a textual or symbolic way, by means of manuals, norms, formal documents, and thus easily found and stored [4, 19].

Knowledge is in constant conversion, going from tacit to explicit, from explicit to tacit, and also being transformed from tacit to tacit, and from explicit to explicit [20]. Each of these conversions has a particular definition.

  • Socialization - is the transference of tacit knowledge to another person, in which there is exchange of experiences between individuals, which can occur through observation, imitation and practice [4];

  • Externalisation - is the process of converting tacit into explicit knowledge, usually triggered by dialogue or collective reflection, but can also be the result of individual reflection [10];

  • Combination - occurs through reconfiguration of existing knowledge leading to new knowledge. It is the gathering and systematization of formal knowledge from different sources that are reorganized by separating, adding, combining and classifying explicit knowledge [10];

  • Internalisation - means to take externalised knowledge and make it into individual tacit knowledge in the form of the mental models or technical know-how [4].

Organizations can not create knowledge alone. Tacit knowledge of individuals constitutes the basis of knowledge creation of organizations, and it is fundamental that the organization mobilizes tacit knowledge created and accumulated at the individual level [20]. This tacit knowledge is propagated by the organization through knowledge conversion and condensed at higher ontological levels.

For Nonaka and Takeuchi [20], the ontological dimension of knowledge is composed of individual, group, organization and inter-organization. Creation of organizational knowledge begins in the individual and is disseminated, expanding interaction groups that cross boundaries between sections, departments, divisions and organizations, establishing a spiral process.

Understanding these dimensions ensures better visualization of how knowledge moves in projects and organizations, making it possible to learn from projects that have already been completed. For example, in the postmortem analysis at the epistemological level, two conversions are widely used: socialization and externalization [10]. At the ontological level, postmortem analysis facilitates dissemination of knowledge from the individual level to the organizational level.

3 Research Method

For the development of this paper an exploratory research was carried out by means of a bibliographical survey on the topics Postmortem Analysis and Knowledge Management. The motivation was to gather information on the subject and delimit the field to be studied.

In examining the subject, it has been identified that postmortem analysis in projects is often neglected [1, 11, 12, 14]. However, the difficulties that led to its non-achievement are often neglected in these articles.

Given the perception of this gap, the main objective of the research is to understand and map what are the difficulties that inhibit the accomplishment of this type of analysis. Knowledge of these difficulties makes it possible to mitigate risks when executing postmortem analysis, a practice that allows improvement of organization and competitive advantage through lessons learned.

Three knowledge bases were used: ACM Digital Library, IEEE Xplore and ScienceDirect, as well as Google Scholar as a tool to increase search comprehensiveness.

With the objective of mapping only recent articles that would bring current experiences, only articles published starting from year 2012 are considered. However, the established criterion was disregarded because few articles - only nine - meet this restriction.

The difficulties found in the articles were categorized into two types: technical difficulties and managerial difficulties. Separation into two types of difficulties expanded the discussion on the subject, because solving the technical part is only part of the overall solution, both difficulties influence each other over time. Managerial support through elements of organizational, operational structures, and engaging communication is critical to organizational learning.

This division made possible a better understanding of the difficulties. These are presented and discussed in the following sections.

4 Technical Difficulties

The main technical difficulties to perform postmortem analysis are presented in this section. At the end, Table 1 presents a summary of technical difficulties mapped from literature.

4.1 Lack of Standards for Performing the Analysis

In summary, the postmortem analysis process consists of four phases: data collection, workshop meeting, data analysis and present results [1]. Several papers that aim to improve this form of analysis in the projects were published [1, 5, 9, 10].

However, in the literature, the lack of methodological support [14] and the lack of an effective method that will yield good results without the need of external consultants or experts [5], are cited as difficulties to carry out this activity.

As presented, there are papers that propose improvements for the postmortem analysis, however some articles [5, 14] indicate that the lack of standards is still a problem. It is suspected that the proposed processes fail to meet the specificities inherent in some types of projects or particularities of the team. Thus, a standardization of tools, techniques, and processes would help both the team and the organization, since the necessary steps would be known to each team member.

4.2 Collection and Data Extraction

Few aspects of the knowledge that is generated during a project is made explicit through documentation. It requires the knowledge that those involved have of the project. However each individual has only the vision of a part of the whole, this can produce incorrect conclusions about the project [3, 14]. For example, in a software development team, there are several roles, such as software architect, test analyst and infrastructure analyst, among others.

It is common that each professional specialization is concerned only with their specific activities. This is a common situation, but it can become a problem if professionals do not share their knowledge, when necessary, with other team members.

Ahonen and Savolainen [1] emphasize that the common phases of postmortem analysis processes heavily depend on participation of project team members and the subjective opinions expressed by them. It is important to collect and extract as much as possible of tacit knowledge present in project members.

Growth in the number of distributed development teams creates major challenges for conducting retrospective projects [15]. It is increasingly common for teams to be distributed in different cities within the same country, or even in different countries. Language barriers can not be disregarded, even among professionals with fluency in the same language, as it is known that there are idioms or terms that can cause different understandings.

Performing data collection and extraction is critical to quality analysis, as finding out which practices should be strengthened and which should be abandoned in future projects is not a trivial task in complex systems, particularly on large, lengthy projects [9], resulting from feedback and dynamic, systemic effects [26].

4.3 Useful and Efficient Histories of Postmortem Analysis

When project retrospective results are not used and knowledge is not disseminated among members and teams, those involved become dissatisfied with the process [10]. This makes it difficult to conduct future postmortem analysis because leaders and the analysis itself lose credibility with the team, turning a tool for organizational learning into something meaningless for the organization.

Having historical information that is useful and easily accessed is fundamental to the incentive of its realization [26]. The link between analysis and future projects must be well-understood [9] so that the benefits of performing postmortem analysis can be realized.

Collaborative tools such as wikis, blogs, institutional portals, or decision support systems are relevant to mitigate such difficulty. Development of techniques and tools that seek to solve this problem help both the team, since this will visualize the results of the work spent, as well as the management, since it makes it possible to observe the return of investment occurred in the project.

Table 1. Technical difficulties encountered in performing postmortem analysis
Full size table

5 Managerial Difficulties

Main managerial difficulties to perform postmortem analysis are presented in this section. At the end, Table 2 presents a summary of managerial difficulties mapped from literature.

5.1 Lack of Specific Schedule for Conducting the Analysis

The greatest difficulty presented in papers is insufficient time [1, 3, 5, 10, 12, 14, 26]. Because it is an activity performed when a phase ends or when a project is terminated, postmortem analysis in many cases does not have a dedicated timetable for its accomplishment, and if it does, it ends up being deferred in detriment of other activities to be completed due to eventual delays in schedule.

Project team members frequently do not have time for meetings, or for sessions to review lessons learned [5, 10], and they happen to be reallocated to other projects before they are done. Glass [12] claims that professionals in the field of Software Engineering are constantly busy and they rarely have time to think about how development could be going better, not just faster.

Taking time to perform postmortem analysis is important because most processes use methods that require active participation of project staff or their combination with the use of project documentation [1].

However, Bjarnason et al. [3] warned that even if project members take the time for a retrospective, it can be hard to correctly remember and jointly discuss past events in a constructive way. It is important to use a schedule with activities carried out in the project to guide the meetings and participation of members.

Since the results provided by postmortem analysis are beneficial to the organization as a whole, using simplified or lightweight version can be a possible solution when time is a constraining factor [5]. However, the results may fall short.

5.2 Lack of Agreement Regarding the Criteria to Be Evaluated

In carrying out the analysis it is necessary that the project criteria that will be evaluated are presented in a clear way. The workshop facilitator should state the objectives of the meeting, if possible, describing the process to be followed [10]. Clarity of the process mitigates risks of meetings being unproductive and having an ineffective environment for sharing lessons learned.

This risk occurs because in organizations that do not have the culture of retrospective project, those involved may feel threatened to share poor project experiences, generating an evaluation and critical environment, rather than learning and sharing. Responsible management should make it clear that the goal is to analyze the process as a whole, not people in an individualized way.

Another risk is that members of the meeting emphasize only the negative aspects of the project, forgetting to strengthen the good practices that have occurred to be repeated in future projects.

A well-defined process of carrying out the postmortem analysis, its phases and criteria, allows those stakeholders to feel secure and protected to share the learning that occurred during the project [9], enabling the conversion of tacit knowledge into explicit and disseminating it through the organization.

5.3 Conflict Between Stakeholders and Information Bias

When the organization does not have a culture of knowledge sharing, conflicts between stakeholders may occur and information provided may hide project events.

One of the concerns in conducting postmortem analyzes is the honest sharing of what happened in the project and the experiences gained. But in projects that have failed, there is a natural disincentive within the organization to conduct a postmortem analysis, it also creates apprehension for individuals preparing to participate in the meeting [1, 9], causing them to become defensive or seeking guilty. Retrospective of the project can turn into an emotional outburst, rather than a constructive discussion on how to improve practice.

It is important to emphasize that postmortem analysis is based on personal experience in events that occurred in the project. The risk of incorrect conclusions is present, because considerations made by those involved observe only a part of the whole that is the project [3, 14]. This vision of the project may be biased even because some team members may not want to participate in the process because they do not wish to do a self-assessment.

To create a good environment for conducting an analysis where there are no distortions and biases in members’ responses, Dingsøyr [10] suggests that managers should not participate in the project retrospective team, because beyond activities needed for the retrospective will be assessing individual performance of those involved, which may prevent lessons learned from developing the project. In this way, the retrospective meeting should be clearly separated from any personal performance assessment [10].

5.4 Lack of Management Support

In order to carry out postmortem analysis, it is essential that there be management support for its implementation. Pressures for delivery of results [9], too busy teams [12, 14], lack of motivation to invest financial, human and time resources in closed projects [26], immediate reallocation from one project to another without time to discuss what was learned during implementation [3] may make it impossible to create retrospective. Due to the lack of support, over time the details and sequence of events are forgotten, losing an important input for organizational improvement [3].

Even if in organizations there is a process for postmortem analysis, if it is not supported by management, it is seldom used in practice [10], and when its results are not used and knowledge disseminated, those stakeholders begin to show dissatisfaction with the process [10], being only a bureaucratic step.

Table 2. Managerial difficulties encountered in performing postmortem analysis
Full size table

6 Conclusion

This article had as objective to identify and to indicate which are the difficulties for accomplishment of postmortem analysis. Through a bibliographical research a set of difficulties were identified and classified into two categories: technical difficulties and managerial difficulties. This classification allowed better understanding of the issue and the proposals that overcome such difficulties.

This paper identified that, even though there are some processes to perform postmortem analysis, the lack of better methodological support prevents this practice. It also clarified that in a retrospective, tacit knowledge is what brings additional learning to organizations. Therefore, ways to capture this knowledge must be well executed and encouraged. It also showed that use of faster and more efficient means of accessing the information generated in postmortem analyzes motivates achievement of the same information both at the operational level and at the managerial level.

In relation to managerial difficulties, the lack of a specific schedule was the difficulty that most appeared in articles. The lack of clear criteria and a well defined process were also mentioned as difficult because they allowed dispersion of those involved and did not direct the meeting to points that bring benefits to construction of knowledge. Conflict among those involved may also be a risk to achievement as it may inhibit participation of members or encourage search for guilty parties. Finally, if there is no support from management for dissemination of this practice, not allocating resources necessary for its realization, it becomes impracticable.

This research provided better understanding of existing difficulties and synthesized this information in two tables. However, a threat to validity is the limited amount of articles that addresses the issue. In order to reduce this threat, the restriction of publication starting from year 2012 was disregarded, considering only the relevance of the published article. This fact even shows shortage of research on difficulties in performing postmortem analysis.

As for future works, a set of templates will be proposed to help individuals to extract information from postmortem analysis of projects. The objective is the standardization in activities of collection and extraction of data, which enables creation of a history that is easily accessible. This work will help to mitigate some of the technical difficulties. Another work to be developed, will be a framework that allows accomplishment of postmortem analysis that can be adapted to reality of a specific organization seeking to reduce the managerial difficulties reported in these works.