1 Introduction

Product-based companies adopt service-led competitive strategies to differentiate themselves (Rothenberg 2007; Bikfalvi et al. 2013). One such strategy is the integration of products and services in a single system, called product–service system (PSS) (Goedkoop et al. 1999; Mont 2002). Examples of PSSs include precise farming solutions (Bettencourt 2010), car-sharing schemes (Williams 2007), chemical management services (Yang et al. 2008), and document management solutions (Rothenberg 2007), which have innovated the way customer needs are fulfilled.

PSSs expose stakeholders to various economic, environmental, and social values. The integration of complementary products and services leads to better customer need fulfillment (Baines et al. 2007), which in turn can contribute to higher company gains. By providing requested services, companies build closer relationships with their customers, enabling them to anticipate future business opportunities and achieve long-term success (Rothenberg 2007; Reinartz and Ulaga 2008). When product deliveries are routed through supplementary services, material consumption and waste emission become more controllable, thereby reducing the adverse environmental impact of consumption (Tukker and Tischner 2006). PSSs provide people with a more balanced propensity to consume, enabling them to make mature choices (Kang and Wimmer 2008). In short, PSS could be an alternative solution in achieving sustainable growth.

PSSs have become a major trend among product-based companies, and its development has emerged as an important agenda to industry and academia (Park and Yoon 2015). A successful PSS development project requires evaluation, which must be performed in a timely and appropriate manner. Evaluation increases the likelihood of success of PSS development and suggests directions for improving an already developed PSS. This study is motivated by the need to aid managers in PSS evaluation (van Halen et al. 2005; Lee et al. 2012).

PSS evaluation requires multiple perspectives on a case-by-case basis, because PSSs are complex combinations of various components and multiple attributes (Tukker and Tischner 2006; Yoon et al. 2012). The research question of this study is as follows: What aspects of the PSS should be examined in its evaluation and how can a holistic evaluation of the PSS be developed? Existing studies on PSS evaluation can be classified into four groups according to the focus of evaluation, namely, economic, environmental, social, and customer values (see Sect. 2 for further details), which must all be considered in PSS evaluation to understand the strengths and weaknesses of PSS. However, no study has integrated the four values into a single framework.

A PSS model demonstrates the functions offered by a PSS, the ways such functions are offered, and the associated stakeholder network. For PSS model evaluation, this paper proposes a comprehensive scheme that considers the economic, environmental, social, and customer values. The proposed PSS model evaluation scheme consists of a set of 94 evaluation criteria and an evaluation procedure (see Sect. 3 for further details). Such wide coverage of evaluation criteria distinguishes this work from existing studies. The set of 94 criteria covers provider and customer perspectives as well as five phases of a PSS lifecycle, namely, design, production, sales (or purchase), usage, and disposal, which are all considered key factors of PSS analysis in the literature (Aurich et al. 2006).

Using the proposed scheme, designers can identify the strengths and weaknesses of a PSS model, and determine directions for its improvement upon complete and effective evaluation. Section 4 presents three case studies that tackle three PSS model types, namely, the existing model, the newly launched model, and the model under development, which were conducted to validate the proposed scheme. The case studies demonstrate the applicability and power of the proposed scheme. The contributions and future research directions of this study are discussed in Sect. 5.

2 Review of related literature

Table 1 shows a list of related works on PSS evaluation. The second row is a collection of works that specifically focus on PSS evaluation. The works listed in the bottom row evaluate products or services. Although these works do not specifically concern PSS evaluation, some of the approaches from these works can be employed for PSS evaluation.

Table 1 Existing works related to PSS evaluation
Full size table

Table 1 also shows the two main perspectives involved in PSS evaluation, namely, sustainability and customer value. Sustainability has three dimensions, namely, economic, environmental, and social values, which are collectively referred to as 3P (profitability, planet, and people). Most works on sustainability focus on one (e.g., Tasaki et al. 2006) or two (e.g., Rothenberg 2007) dimensions. Works that cover all three dimensions (e.g., van Halen et al. 2005) are scarce.

A large portion of the results in Table 1 is focused on sustainability, and only a few have explored customer value. The dimensions of sustainability represent the perspective of the PSS provider or the entire society. By contrast, customer value is derived from the perspective of individual customers. The customer value perspective is also important because it helps fulfill customer needs through the integration of products and services in the PSS (Tukker and Tischner 2006; Lim et al. 2012). Rese et al. (2009) suggested a method for measuring customer value in terms of net present value.

Three observations were made from the literature review. First, few works considered both provider and customer perspectives in PSS evaluation, with most works focusing only on the former. The evaluation scheme should include both provider and customer perspectives because of the significance of customer satisfaction. Second, few works considered all the 3P for sustainability. PSS evaluation should establish a balance among all the 3P values to facilitate inclusive growth. Third, the PSS lifecycle has not been comprehensively discussed in existing works. Only a few have considered the lifecycle aspects (e.g., Vogtlander et al. 2002; Pecas et al. 2009). PSS value is perpetually created throughout its lifecycle (Aurich et al. 2006). Therefore, the various phases of a PSS lifecycle should be considered in the evaluation scheme.

3 Proposed PSS model evaluation scheme

The PSS model evaluation scheme consists of evaluation criteria and an evaluation procedure. Section 3.1 describes the proposed set of 94 evaluation criteria and the research process for determining the current set. Section 3.2 introduces the evaluation procedure.

3.1 PSS model evaluation criteria

Table 2 presents the four-layer hierarchical structure within the proposed evaluation criteria. The uppermost layer of the structure is the perspective level, which refers to the global viewpoint of the evaluation. Each perspective has several dimensions, and each dimension has categories. Dimensions represent particular portions of a perspective. Categories refer to the broad areas of issues related to the corresponding dimension. A brief explanation of each category is provided in Table 2. Each category has several items that refer to the specific types of information related to a given category. The column farthest to the right in Table 2 supplies the references related to each category.

Table 2 Structure of the PSS model evaluation criteria
Full size table

The 94 evaluation criteria were developed as follows. First, evaluation criteria and other findings were collected from existing PSS evaluation studies (e.g., van Halen et al. 2005; Omann 2007; Garvin 1984; Guinée 2002; Parasuraman et al. 1985). Table 2 distinguishes related articles that explicitly propose some evaluation criteria (denoted by *) from those that discuss relevant content (without *). Second, the collected criteria and findings were categorized into the five dimensions, namely, the 3P (provider perspective), quality, and cost (customer perspective). In this process, the criteria proposed by existing studies were adopted and/or adapted as determined appropriate to the categorization rule. Third, the developed criteria were refined with regard to how effectively such criteria provide various viewpoints for evaluating PSS cases in our case repository (Kim et al. 2012; Lim et al. 2012). The repository stores 181 PSS cases (123 product-oriented, 35 use-oriented, and 23 result-oriented types) collected from journal articles, books, conference proceedings, news, and PSS-related websites. Lim et al. (2012) provides a more detailed explanation of the repository. During this refinement, the authors also developed new evaluation criteria based on our experience in PSS development projects with industries and relevant studies. Finally, the authors modified (e.g., combined, divided, or renamed) the evaluation criteria as necessary. The authors repeated this process several times, ultimately leading to the 94 PSS evaluation criteria listed in Table 2.

The five-digit prefix of a dimension, category, or item in Table 2 represents the code for identification. The perspective layer consists of two perspectives, namely, sustainability (provider) and customer value (customer). The 3P are considered in the sustainability perspective. The profitability of a PSS model is evaluated in terms of fixed and operational costs, revenue, ecosystem structure, and macroeconomic effects. The planet dimension evaluates environmental value by considering product, material, and energy usage; emissions of toxic substances; and environmental management. The people dimension evaluates social value based on the capability of employees, profit sharing, working environment, employment equity, acceptability, and influence on society.

The PSS quality that customers perceive and the cost that customers pay are considered in the customer value perspective. PSS quality is determined in terms of the quality of the product or service and other quality-related issues, such as customer support and system convenience. PSS cost includes the amount paid by customers for the purchase, usage, and disposal phases of a PSS.

The proposed evaluation criteria were designed to consider both the provider and customer perspectives, the 3P (profitability, planet, and people) dimensions, and the phases of a PSS lifecycle. No existing study has fully discussed customer value or the characteristics associated with different PSS lifecycle phases. Thus, the main advantage of the proposed scheme is its ability to explore comprehensive coverage of the evaluation criteria.

3.2 PSS model evaluation procedure

The PSS model evaluation procedure involves (1) defining the evaluation scope, (2) selecting the relevant evaluation criteria, (3) scoring, and (4) aggregating scores. A formal and definite evaluation scope is a prerequisite to the evaluation of any PSS type (Step 1). PSS visualization is a useful method for accomplishing this objective. This method aids in the effective and clear comprehension of a PSS as well as of the stakeholders and components involved. Thus, visualization helps identify the evaluation objects and their relationships systematically.

Existing PSS visualization frameworks are classified according to the focus of visualization, namely, PSS process and network visualization. PSS process visualization frameworks support users to depict a series of information related to the PSS provision in chronological order. Examples of such frameworks include Extended Service Blueprint (Hara et al. 2009), Product–Service Blueprint (Geum and Park 2011), and PSS Board (Lim et al. 2012). If the visualization purpose is to analyze the value creation via information exchange between PSS provider and customers, Information Service Blueprint (Lim and Kim 2014) would be useful for visualizing a PSS process. PSS network visualization frameworks are useful to visualize the relational network of PSS stakeholders and the aspects involved in the relationship. Examples of such frameworks include System Map (van Halen et al. 2005), Interaction Map (Morelli 2006), and Relation-based Model (Kang et al. 2011).

Step 2 is the selection of relevant evaluation criteria. All of the criteria in Table 2 may not be relevant to the evaluation of a particular PSS model. Only the criteria relevant to the evaluation scope of the given PSS model are selected and utilized. Assessment of relevance may vary depending on the judgment of PSS evaluators. PSS evaluation is, by nature, a “soft” task that requires human activities. PSS evaluation should consider various perspectives because the objective of a PSS evaluator depends on his/her functional unit. An evaluator from the finance department may be primarily concerned about profitability, whereas an evaluator from marketing may focus on customer value. Incorporating various perspectives into the evaluation process increases the likelihood of identifying various improvement points of PSS models. Thus, the key in Step 2 is not to be strict in assessing relevance and to integrate different perspectives for a comprehensive and balanced evaluation. The contribution of the proposed evaluation scheme is to provide a basis for this purpose.

In Step 3, the PSS model must be scored using the selected evaluation criteria. A five- or seven-point scale is widely used for such scoring because of its simplicity. More advanced scaling methods, such as confusion, partition, and ratio scaling (Gescheider 1997) can be used if desired.

For Step 4, weighted-sum approach is a popular method for aggregating multiple criteria into a single representative value. This approach determines the relative weights of the criteria and then sums their weighted values. Several methods, such as analytic hierarchy process (Saaty 1977) and multi-attribute utility theory, (Keeney and Raiffa 1993), can be used for weighting the criteria. Statistical methods such as regression, structural equation modeling, and factor analysis (Neter et al. 1996) can be used as well.

Several methods can be used in conjunction with the proposed PSS model evaluation scheme. Quality function deployment (QFD), which is an effective method for developing new products, uses a weighted-sum approach for evaluation (Hauser and Clausing 1988). Kobayashi (2005) and Shimomura et al. (2008) applied QFD on an eco-product design and service evaluation, respectively, while Kim and Yoon (2012) adopted QFD to identify the critical features of products and services. Data envelopment analysis (DEA) is another potentially useful method for PSS model evaluation. DEA was originally developed for measuring system productivity (Charnes et al. 1978), but it can be employed to compare the relative efficiency of PSS models in terms of the 3P. Park and Tahara (2008) suggested a DEA method for measuring the eco-efficiency of designs.

4 Case studies

Three case studies were conducted to validate the proposed evaluation scheme. The first study evaluated three automobile-related PSS models, namely, car leasing, car sharing, and carpooling models. The second evaluated a smartphone-based enterprise resource planning (ERP) service called Smart-CEO, which is provided by Company S, a telecommunications company in Korea. The third evaluated a telematics-based car maintenance service of a car manufacturer in Korea. The main objective of the first case study was to test the workability of the proposed evaluation scheme by illustrating the use of the proposed scheme in evaluating an existing PSS case. The second case study evaluated the ability of the proposed evaluation scheme to provide useful insights on a newly launched PSS case. The third case study assessed the applicability of the proposed evaluation scheme in evaluating a PSS model under development.

4.1 Case 1: evaluation of automobile-related PSS models

Three PSS models were evaluated, namely, car leasing (Model 1), car sharing (Model 2), and carpooling (Model 3). The three models are distinct in terms of the role of the PSS provider. In Model 1, the PSS provider leases cars to customers. In Model 2, the PSS provider maintains and provides cars to be shared by customers. In both models, the PSS provider owns the cars. By contrast, the PSS provider in Model 3 does not own the cars and only acts as an intermediary. Communication among customers (i.e., car owners and people who need a ride) is supported by a secured network.

Evaluation was conducted using the evaluation procedure introduced in Sect. 3.2. First, the authors visualized the three models using PSS Board (Lim et al. 2012). PSS Board is useful for defining the evaluation scope, because it visualizes how the four components of PSS, namely, product, service, dedicated infrastructure, and provider network, contribute to achieving customer goals. Figure 1 shows the visualized car-sharing model on PSS Board.

Fig. 1
figure 1

A car-sharing scheme visualized on PSS Board

Full size image

The defined PSS models were then scored using relevant evaluation criteria. Only the criteria relevant to this specific case study were selected from the evaluation criteria in Table 2. Thus, 20 out of 21 categories and 82 out of 94 items were used in this study. Similar to Pugh’s concept comparison method (Pugh 1996), the three models were evaluated against a reference model. The conventional car selling model was set as the reference model, denoted as Model 0. A five-point scale (much worse, worse, the same, better, or much better than Model 0) was used to score each evaluation item. These five levels were assigned the scores of −2 to +2, respectively, to quantify the evaluation results (i.e., zero represents the same). Evaluation was performed based on the information gathered from various sources, including an existing study on PSS models in the automobile industry (Williams 2007) and the Internet. The scores were averaged to derive the scores of their corresponding category once the evaluation scores of the items were obtained. The average scores of the categories were then computed to derive the scores of the corresponding dimensions.

Figure 2 shows the evaluation scores at the dimension level. Overall, Models 1 and 2 did not significantly differ from Model 0. Model 3 was clearly better than Model 0 in terms of planet and cost dimensions. Evaluation results can be checked at the category level to obtain a more detailed information. In Fig. 3, the categories of energy use and emission of toxic substances clearly contributed to the high score of the planet dimension in Model 3. The evaluation of Model 3 also revealed its weaknesses in the people and quality dimensions. The social atmosphere (e.g., social network infrastructure, trust, and safety) must be mature, so that Model 3 can run smoothly. The quality of a carpooled ride is difficult to guarantee because the PSS provider does not own and maintain the cars. The introduction of customer behavior (pertaining to both the driver and the rider) and car maintenance monitoring systems may prevent such potential problems and improve the PSS model.

Fig. 2
figure 2

Evaluation results of automobile-related PSS models at the dimension level

Full size image
Fig. 3
figure 3

Evaluation results of automobile-related PSS models at the category level

Full size image

4.2 Case 2: evaluation of Smart-CEO PSS model

Smart-CEO is a mobile service that provides useful management information (e.g., daily sales, revenue, profit, expenditure, and inventory) extracted in real time from the enterprise resource planning (ERP) system of a company. A Smart-CEO user can make management decisions anytime and anywhere, and then immediately spread such decisions to other employees. Smart-CEO targets high-level executives of small- and medium-sized companies that utilize an ERP system. Smart-CEO can be considered a PSS because various service functions are provided in mobile devices such as tablet PCs and smartphones.

Smart-CEO was only recently launched in the market at the time of this evaluation. Company S intended to extend the line of business if Smart-CEO succeeds in the domestic market, suggesting that Smart-CEO could be a potential leader in the global B2B solution market. For this reason, Company S wanted to evaluate the attractiveness of the Smart-CEO model.

First, the authors defined the Smart-CEO model using a PSS network visualization tool called relation-based model (Kang et al. 2011). Among the evaluation criteria in Table 2, 10 out of 21 categories and 34 out of 94 items were selected for evaluation. The planet and people dimensions of sustainability were considered irrelevant to the purpose of Company S and were thus excluded in the evaluation. The conventional usage of an ERP system through a desktop device called “Desktop ERP” was set as the reference model. As in Sect. 4.1, an evaluation of Smart-CEO against Desktop ERP was conducted using a five-point scale. The evaluation was performed based on the information gathered from face-to-face interviews with the manager of Smart-CEO and two other managers from related divisions in Company S.

Figure 4 presents the evaluation results at the category level. Overall, Smart-CEO is better than Desktop ERP in terms of profitability (average score of 0.21) and quality (average score of 0.45). However, the two models received the same score on cost (average score of 0). The high score in profitability was attributed to the high potential of revenue creation despite the initial capital outlay needed for launching and service development. The excellence in quality was derived from the usefulness of the information provided by Smart-CEO. Quality was also high because of the convenience of the delivery process of Smart-CEO.

Fig. 4
figure 4

Evaluation results for the Smart-CEO PSS model at the category level

Full size image

Smart-CEO exhibited weaknesses in terms of fixed/operational cost and ecosystem structure categories. The low score on fixed/operational cost was expected because Smart-CEO is still at its early stage. However, the problem of ecosystem structure is critical to its success and must be remedied. This problem usually generates restricted or harmful internal flows of information, material, cash, and so on. Thus, additional discussions were conducted with counterparts at Company S regarding the structure of the Smart-CEO ecosystem.

Figure 5 presents the structure of the Smart-CEO ecosystem. At present, Smart-CEO is only available to users of the ERP system from Company D. That is, the ERP system of Company D is a prerequisite for using Smart-CEO. Therefore, the current Smart-CEO ecosystem possesses a closed structure that relies on a particular participant, Company D. Consequently, Company S can provide only a limited range of services to a restricted number of customers. Company S was thus advised to acquire and integrate other ERP system providers to its network, which would lead to a healthier ecosystem and an expansion of the available service menu.

Fig. 5
figure 5

Structure of the Smart-CEO ecosystem

Full size image

4.3 Case 3: evaluation of a telematics PSS model

A car manufacturer in Korea harvests data on vehicle operations and health through a telematics system and has constructed a database called Vehicle Relationship Management Database (VRM DB). The company has tried to develop attractive new telematics services using VRM DB. The authors participated in a project to develop such services. This paper introduces the evaluation of a telematics service under development. Vehicle consumables are usually replaced in a fixed-unit manner (e.g., every 10,000 km or 6 months). The new concept, called data-driven consumable replacement support service, decides the replacement time based on an analysis of VRM DB. If the driving patterns of customers are severe and detrimental to the consumable, the recommended replacement period decreases from the standard.

The evaluation process of the data-driven consumable replacement support service was similar to that for the other two case studies. The authors identified various weak and strong points of the PSS model and compared it with the conventional car maintenance model. The authors suggested a data collection and management system upgrade to the company. This upgrade would enhance the understanding of customer behavior and vehicle health, and subsequently prepare more data-driven service concepts in a step-by-step manner.

Basing on Case study 3, the authors identified three differences between the evaluations of PSS models under development and those already launched into the market. First, information availability is limited in the new PSS model development environment because the degree of model completion is not high. As such, the evaluation criteria in Case study 3 were identified mainly at the category level. However, improvement ideas can be identified even with category level criteria. Second, the speed of evaluation is quicker because the evaluation can be accomplished in a laboratory. In fact, quick evaluation and decision-making processes are desirable in new PSS development projects. Third, an iterative process between design and evaluation can be achieved. The feedback gathered from evaluation provides designers with various ideas for improvement. The improved model can be further refined through subsequent evaluation.

5 Concluding remarks

Despite its importance, PSS evaluation has not been extensively explored in the literature. This paper proposes an evaluation scheme for PSS models. The proposed scheme has a four-layer hierarchical structure of evaluation criteria consisting of two perspectives (sustainability and customer value), five dimensions (profitability, planet, and people for sustainability; quality and cost for customer value), 21 categories, and 94 items. The proposed scheme also considers various phases of a PSS lifecycle, namely, design, production, sales (or purchase), disposal, and usage. The proposed scheme involves four steps: defining the evaluation scope, selecting the relevant evaluation criteria, scoring, and aggregation of scores.

This research has theoretical and practical contributions to the PSS literature. From a theoretical perspective, the proposed scheme integrates several work streams within the evolving PSS literature in consideration of PSS evaluation. The integrated studies include the 3P dimensions of PSS, product and service quality, customer values, PSS lifecycle, and PSS cases. This integration is warranted because a comprehensive and balanced PSS evaluation should consider diverse viewpoints. The proposed PSS evaluation scheme provides an integrated knowledge base in an organized manner.

From a practical perspective, this research attempts to provide systematic support to PSS evaluation. The use of the proposed evaluation scheme in a real project environment has three advantages. First, the proposed scheme is sufficiently comprehensive to cover provider and customer perspectives as well as lifecycle phases of PSS. Second, the proposed evaluation scheme can be used in various stages of PSS development by selecting the appropriate criteria for the object to be evaluated. The proposed scheme can also be employed to evaluate PSS ideas, models, or completed PSSs. Third, the evaluator can utilize the wisdom from the experience of past PSS evaluations rather than starting from scratch. Consequently, the efficiency and effectiveness of PSS evaluation will increase.

These advantages were validated by the case studies in Sect. 4. The strengths and weaknesses of PSS models were systematically identified by applying the proposed evaluation scheme in the case studies. The proposed scheme simplified the evaluation process by extending the initially limited frame of reference. Ideas and issues for improvement were also found. The proposed evaluation scheme is workable and comprehensive enough to cover broad aspects, and it can provide useful and balanced assessment of the potential value of the PSS models in question. The proposed scheme shares the same philosophical basis as the balanced score card (BSC) (Chang et al. 2013). As BSC has contributed to measuring business unit performance, the proposed scheme will support its users (e.g., PSS managers and researchers) to achieve a balanced assessment of the potential value of the PSS models in question. The proposed scheme should be validated further in various settings, but its potential to grow as a generic platform for PSS evaluation was demonstrated.

To improve the evaluation scheme, several issues for future work are suggested. First, the comprehensiveness of the criteria should be repeatedly checked and updated over time. Real PSS cases should be collected and evaluated using the scheme. The developed evaluation scheme can be generally applied to any type of PSS, but conducting a case study of a B2C-type PSS would be interesting because it explicitly considers the “experience” of customers. A product- or use-oriented PSS (where the responsibility of the customer tends to be widespread over the lifecycle) can be effectively evaluated because the evaluation scheme encompasses the lifecycle of a PSS. This evaluation scheme will become more reliable and realistic as more cases are tested. Second, the accompanying evaluation methods and procedure should be strengthened. The evaluation methods should include various decision analysis and support tools. Methodologies for multiple criteria decision-making (e.g., Ho et al. 2010) would create great synergy with the proposed PSS model evaluation scheme. The evaluation procedure should provide a detailed guide for conducting an evaluation. Third, a feedback mechanism after an evaluation should be developed. The evaluation results should be reflected and utilized in devising a PSS improvement plan. Detailed guidelines that discuss the aspects of the PSS, which are subject to refinement and revision, should be constructed when evaluation scores are unsatisfactory.