Keywords

1 Introduction

The goal of maintenance sustainability assessment is to provide information on the current maintenance performance and support decision-makers in the decision-making process regarding future directions of operations [1,2,3]. This information should be synthetic, and thus show the result of the assessment in an aggregated way, and at the same time enable decomposition to lower levels showing the impact of each of the assessed criteria on the result.

One of the methods of developing performance measurement models most frequently mentioned in the literature, from the perspective of sustainable development is a balanced scorecard (BSC) developed by [4]. From the point of view sustainability assessment this model has four important features: (1) combines the strategy with the objectives and measures of their implementation; (2) includes and links financial and non-financial measures; (3) considers links between internal effectiveness of processes and their external efficiency; and in addition; (4) enables inclusion of dimensions of sustainable development. Since the BSC was introduced, many authors have proposed modifications for adapting the initial BSC to other models that are specific to different areas or industrial environments. The application of the balanced scorecard in tracking maintenance action plan effectiveness was reported in [5], who mentions in his study the use of the balanced scorecard as a medium for educating maintenance personnel on the organization’s maintenance strategy. Adapting from the original BSC, [6] coined the term “maintenance scorecard” (MSC). He defined the MSC as an approach used to develop and implement strategies for the area of asset management in both short and long terms, and defined six areas of importance of asset management: (1) Productivity Perspective; (2) Cost Effectiveness Perspective; (3) Safety Perspective; (4) Quality Perspective; (5) Environmental Perspective; (6) Learning Perspective. The MSC is used to develop and implement a strategy in the area of asset management. It also serves to identify strategic improvement initiatives, along with the areas they focus on, early in the process. The extended BSC presented in [7] incorporates performance measures based on seven perspectives: corporate business (financial), society, consumer, production, support functions, human resources, and supplier perspectives. In [8] the authors suggest a performance management framework based on the BSC model and a list of key indicators for a project for the Norwegian oil and gas industry. The framework considers cost, operation, HSE, and organization perspectives. Maintenance and employee satisfaction are not included. However, in the work [9] a multi-criteria hierarchical framework for MPM that consists of multi-criteria indicators for each level of management, i.e. strategic, tactical and operational is proposed. These multi-criteria indicators are categorized as equipment/process related (e.g. capacity utilization, OEE, availability, etc.), cost related (e.g. maintenance cost per unit production cost), maintenance task related (e.g. ratio of planned and total maintenance tasks), customer and employee satisfaction, health, safety and environment (HSE). Indicators are proposed for each level of management in each category. In the work [10] authors developed a maintenance performance measurement model using three reference models - the Cost of Poor Maintenance Model, the Malcolm Baldrige National Quality Award and the Context-Input-Process-Product assessment model. Based on their research results, they identified the most important factors affecting the results of maintenance and assigned them to four classic BSC perspectives: learning and growth, internal process, customer, and finance, and then identified the corresponding indicators. The developed model was validated on the basis of a case study in a real company. In the paper [11] authors based on the BSC model, developed an original structure for evaluating sustainable maintenance performance for automotive companies which consists of eight perspectives assigned to three dimensions of TBL: (1) economic: cost effectiveness perspective, quality perspective, productivity perspective; (2) environmental: environmental perspective; (3) social: learning and growth perspective, health and safety perspective, employee satisfaction perspective, stakeholder’s satisfaction perspective.

The analysis of different models of maintenance results assessment from the point of view of sustainable development presented in the literature indicates that:

  1. 1.

    Most frameworks attempt to address economic performance, but they are still using traditional economic indicators that are not the true measure of sustainability (e.g. spare parts stock price, profitability).

  2. 2.

    Social issues receive the least attention in the existing performance measurement framework. In only a few cases when workers issues are included, they cover mainly health and safety but not worker well-being or job security.

  3. 3.

    Although there are dependencies between assessment perspectives (economic, environmental and social) and indicators, in most of the works links between them are not analyzed.

  4. 4.

    BSC method does not include any techniques for quantifying the synthetic value of all perspectives. This problem could be solved by using for example Choquet integral [12] Moreover BSC method does not include any techniques for quantifying the contribution of each perspective, or criteria/indicators within the same perspective.

The aim of this paper is to proposed maturity model to solve the problem with assessment of each perspective of maintenance sustainability BSC model. This paper is a continuation of the previously undertaken work presented in [12, 13].

Given the purpose above, the paper is organised as follows: in Sect. 2 the overall methodology for aggregate maintenance assessment is presented. Then, in Sect. 3 the maintenance assessment procedure according developed methodology is described. Finally, the conclusions and direction of the future research are presented.

2 Maintenance Sustainability Assessment Model

The maintenance contribution in the realization of sustainability challenges is dependent on the operational and business context of a company. Therefore, in order to support maintenance decision makers in attaining sustainability and to point out the way of maintenance function contribution to sustainable manufacturing, main maintenance factors affecting sustainable manufacturing should be identified and analysed. Due to the need for simplicity and clarity in decision-making support, the information provided to decision-makers in the form of recommendations needs to be unequivocal, logical and easy to interpret. The answer to this problem was given in the form of the Composite Maintenance Sustainability Index (CMSI) [12]. In this paper authors developed performance assessment model, which integrate three sustainability dimensions (economic, social, and environmental) with four Kaplan and Norton’s balance scorecard perspectives (financial, client, internal processes and learning & growth). The model consists of two inference levels, the first one encompassing the assessment perspectives and the other including the assessment criteria (Fig. 1). The detailed description of the perspectives and respective criteria can be found in [12]. The result is a synthetic indicator of performance, based on the paradigm of aggregate assessment. The above-mentioned model approaches the problem of aggregation function, for which its mathematical properties point to formally correct aggregation of criteria and behavioural properties express relationships between criteria including, for example, synergy and redundancy.

The general scheme of methodology for aggregate maintenance assessment includes three main stages: (1) Assessment criteria selection, (2) Selection of criteria assessment methods, and (3) Development of Composite Maintenance Sustainability Index (CMSI) (Fig. 2).

Fig. 1.
figure 1

Hierarchical model for maintenance sustainability assessment [8]

Full size image
Fig. 2.
figure 2

A generic process for calculating CMSI

Full size image

The model of sustainable maintenance performance assessment developed according to the three stages scheme (Fig. 2) should help maintenance managers put the strategy into action and offer predictive measures for future performance. To apply the model, it is needed to collect data and assess each of the criterion. The assessments are then aggregated using the Choquet integral [12]. The calculated CMSI value can be then used to determinate the relative importance between perspectives and criteria. Such procedure of calculation of CMSI value can be helpful for decision-makers to pay their attention to the areas that need improvements.

In this paper the stages of the criteria assessment method will be detail presented. The development of the criteria assessment method (Fig. 1) requires consideration of two aspects. First, one should determine the way data will be obtained and the method of their evaluation (e.g. index, descriptive, point). Secondly, the way in which aggregations of partial assessment should be defined within each criterion.

3 Method of Maintenance Criteria Assessment

Because the result of maintenance assessment is a function of quantitative and qualitative variables, in the process of evaluating each criterion, it is necessary to use information and data acquisition tools such as: review of documents, databases and methods of their collection and supervision, direct observation of events, interview with staff (in the assessment process, information from people involved in technical support is very important if they are properly confirmed by objective records). An adequate tool for obtaining data to assess criteria in the model is, therefore, a maintenance audit. In the paper [14] authors defined a maintenance audit as an ‘examination of the maintenance system to verify if the maintenance management is carrying out its mission, meeting its goals and objectives, following proper procedures, and managing resources effectively and efficiently’. According to [15], a maintenance audit enables the integration of two different assessment methods, namely quantitative and qualitative. From the quantitative point of view, it makes it possible to assess the measures and indicators used in maintenance, the purposes of their application and the current status, and thus to examine the difference between the target value and the current one. On the other hand, the qualitative method allows to assess the effectiveness level of activities that are being carried out.

Audit programs consist of key elements that are examined through a set of statements or questions. Each statement or question has a score and a weight. Then based on the audit, a total weighted score is compiled and compared to an ideal score. The scores serve as a foundation for an improvement action plan. The process is repeated periodically to ensure continuous improvement. Considering the above, it is necessary to specify: (1) The scope of the audit program; (2) The subject of research within the scope of the audit; (3) The method of evaluating the subject of research; (4) The method for the aggregation of partial assessment.

  1. (1)

    The scope of the audit program.

From the point of view of the data obtaining method for calculating the CSMI index value, the scope of the audit program is defined by four perspectives of the sustainable maintenance assessment model and by the criteria describing them (see Fig. 2).

  1. (2)

    The subject of research within the scope of the audit

The subjects of research within the defined audit program are detailed issues characterizing each of the criteria of the sustainable maintenance assessment model. Detailed issues were identified based on the analysis of sustainable production requirements, principles of sustainable maintenance and a criteria. An example of the issues under consideration in the ‘Maintenance processes’ perspective are:

  • Analysis and improvement – assessment subject: Improvement system; Analytical methods; Failure investigation; Failure mode analysis; Oil analysis program; Equipment modification; Resource utilization analysis.

  • Implementation and measurement – assessment subject: Reporting procedure (PMn and emergency); Lubrication reporting; Measuring schedule compliance; Quality of PMn inspection; PMn prioritization; PdM work orders creation; Lubrication KPIs; Work Order Closeout.

  • Planning and scheduling – assessment subject: Design of maintenance plan; Identification of E&S requirements; PMn content and procedures; PMn scheduling; Lubrication selection; Lubrication program design; Identification of equipment criticality; Work orders.

  • Management of external service providers – assessment subject: Outsourcing activities; Risk analysis of contractors; Performances of service providers; Principles of cooperation with suppliers.

  • Management of spare parts and consumables – assessment subject: Risk analysis of spare parts suppliers; Performance of spare parts suppliers; Determination of required spare parts; Ordering spare parts and consumables; Storage of spare parts and consumables.

Individual criteria differ in the number of issues assessed, but it seems unreasonable to strive for the harmonization of the number of issues applied to each criterion. Each criterion, because of the scope to which it applies, requires the collection of a different scope of information and data.

  1. (3)

    The method of evaluating the subject of research.

In general, the issues being investigated can be assessed using one of the following methods: indicative, descriptive or a point method. The most popular and internally diversified groups of methods are quantitative indicator methods. They enable the identification, measurement and evaluation of economic and non-economic effects. A large part of the literature proposes useful indicators and metrics for the performance of assessment [11, 16, 17], but does not deal with the problem of data collection. The second group of methods are descriptive methods which are devoid of any formalization elements. They recognize and value qualitative/quantitative characteristics of the assessed phenomena by way of logical analysis and presentation of the test result in a descriptive form. The third group of methods is point methods. Their use identifies measures and values both measurable and verbal qualities.

In the proposed model of sustainable maintenance assessment, a point method was selected to assess the issues describing individual criteria. In comparison with the other two methods (indicative and descriptive), this method has three basic advantages. Firstly, is simple to use. Secondly, the values of features in the point method are expressed in homogeneous, non-quantified numbers (grades of the adopted point scale), which makes it possible to aggregate partial grades into a synthetic evaluation, without the need for their prior normalization and standardization. Thirdly, the point method, apart from the main objective, which is the valuation, provides additional information on the level of implementation of requirements for a given issue. This information may constitute a significant support for the designers of improvement activities. Nevertheless, this method has also its drawbacks. Many problems appear when choosing the right span of the point scale. Literature studies indicate that the spread of the rating scale should not be less than three levels and not more than ten. With regard to the assessment of maintenance, this scale should express levels of maturity adequate to each of the issues assessed. Therefore, in order to assess the issues that characterize each perspective, appropriate maturity models should be built. Maturity models can be used both as an assessment tool and as an improvement tool [18]. Maturity models allow to evaluate the maintenance system and its processes in accordance with good practices. That models are focused on behaviours and thanks to this, allow to identify the next steps that should be taken to reach higher maturity levels [19, 20]. The identification and characterization of maturity models and maturity levels have been discussed in [20,21,22,23,24]. Taking the above and that the data are obtained by maintenance audit, the issues to be assessed will be represented by statements or questions, and answers may take one of the following forms: (1) selecting ‘yes’ or ‘no’ or (2) putting an item on the Likert-type scale to reflect different levels of meeting the requirement. Both forms of response representation require the development of an adequate point scale. Based on [14], a 5-point scale (maturity levels) was adopted, where ‘0’ means that no action was taken, while ‘4’ means that the issue is fully implemented. If the issues are formulated in the form of a question, and the evaluators will be able to choose the answer ‘yes’ or ‘no’, the answer ‘yes’ will be the highest possible number of points, i.e. ‘4’, whereas the answer ‘no’ the lowest ‘0’. In other cases, the evaluators will have to choose one of the ordered and uniquely characterized categories (standard values) by assessing the issue (statement or question). A specific number of points from the scale will be assigned to the categories. The reference values will be described using qualitative characteristics for each of the proposed issues. Figure 3 presents the developed assessment matrix for the ‘Maintenance processes’ perspective on the example of analysis and improvement criterion. The structure of assessment defined in the above manner will allow for a common language of communication while discussing the current situation and planning the future development of the maintenance system amongst interested professionals from various departments in the company (for example, amongst mechanical engineers, production engineers and managers).

  1. (4)

    The method for the aggregation of partial assessment.

The general assessment of each criterion is calculated by the aggregation of partial assessments of issues describing them. Based on the literature analysis, the method used by the Australian Maintenance Excellence Awards [25] was adopted, according to which the general assessment of the criterion is calculated as the ratio of the sum of points obtained for all issues being assessed to the sum of all possible points under the criterion. The value obtained in this way are, on the one hand, input data for calculating the CMSI index (Fig. 4), and on the other hand, they are analytical measures allowing for an in-depth analysis of the maintenance results in individual assessment criteria.

Fig. 3.
figure 3

The developed assessment matrix for the ‘Maintenance processes’ perspective on the example of analysis and improvement criterion

Full size image
Fig. 4.
figure 4

Construction process of the non-additive fuzzy integral for CMSI

Full size image

4 Conclusion

The criteria of maintenance sustainability assessment BSC model can be assessed by an maturity model. The result is a measure of maturity level of each maintenance assessment perspective in BSC model. The main goal is to provide an improvement activities in the maintenance management to achieve sustainability outcomes. In other words, it is a measure of the organizational understanding of, and application of the sustainability challenges of maintenance key-processes or how compliant the maintenance key-processes are with the best practices.

The criteria assessment method presented above meets two functions in the company. First, cognitive, by providing knowledge and possibility of using it for organizational learning. Second, utilitarian, as it allows to create directions of improvement adequate to the current context of the enterprise, paying attention to maintaining balance be-tween economic benefits and environmental and social requirements.