1 Introduction

Cloud computing has been increasingly adopted by a growing number of organizations, due to the advantages such as cost reduction, improvement in efficiency, collaboration, and business opportunities as well as support to green IT, integration and resource consolidation (Buyya et al. 2009; Marston et al. 2011; Weinhardt et al. 2009). Different types of services have been emerged apart from the typical three types: Infrastructure, Platform and Software as a Service for manufacturing. Emerging services for Cloud manufacturing do offer additional incentives such as business agility, since a large number of information can be processed intelligently and presented in a way that users can comprehend with ease. This allows the manufacturing organizations to streamline the process, improve the efficiency and save the costs, since all the work can be completed by Cloud manufacturing services with less time taken and long-term cost-savings achieved (Wilkinson 2008; Williams 2012). Cloud Computing has been used in manufacturing as follows. First, Xu (2012) explain the overall trend, technologies and future direction bridged from Cloud Computing to Cloud manufacturing. Second, Tao et al. (2011) describe their service oriented architecture for Cloud manufacturing with its detailed architecture. However, all these examples do not go deep enough to interpret problems faced by Cloud manufacturing and there is also a lack of data analysis of business performance of manufacturing organizations.

Since more varieties and options of services are on offer, there are challenges that require to be undertaken. First, security and privacy remain a challenge for organizations since all services and data should be secure and protected in real time (Zissis and Lekkas 2012). All the users can be offered the latest security measures and updates to ensure that their privacy and security can be enhanced (Chang et al. 2015). Second, the risk of Cloud Computing adoption exists in all types of the services, including manufacturing (Kagermann et al. 2011). Risk includes operational risk such as uncertainty in the market, customer response and satisfaction and price fluctuation of goods and services (Khajeh-Hosseini et al. 2011; Chang 2014a) which may affect manufacturing. Operational risk can also be divided into financial and customers, since the first type of risk is focused on the impacts to the financial loss and variation in costs of running businesses and the second type is focused on the overall satisfaction of users and customers and whether they will continue their support. Risk also includes the technical risk such as security, loss of data and outage. Details about risk will be presented in literature review. In this paper, it is the risk of Cloud Computing manufacturing adoption (CCMA) that requires to be evaluated. Third, the return on the CCMA should be measured and analyzed periodically. While return on investment can be on the financial aspect to focus on the profits returned on the investment of CCMA, the return in the technical and user focus aspect of CCMA should be considered, since the improvement of efficiency, collaboration, user satisfaction and stakeholders’ trust are important for successful CCMA. In general, all these challenges are interests of investigation for Cloud Business Performance, since all the issues with regard to risk and return can be measured and evaluated while providing Cloud Computing services at the same time.

Cloud Business Performance (CBP) is important for evaluation of Cloud Computing manufacturing adoption (CCMA) as follows. First, stakeholders need to know analysis and reflection about the Cloud Computing services they have agreed and invested. This includes the yearly review about the services they back, in order to justify their decisions of investment are relevant. Second, users of the CCMA can understand the types of risk and return they have in their Cloud Computing services and the extent of their impacts to the services on offer. Third, the organizations involved in the use of CCMA can understand their status of risk and return in details, so that they can learn whether their Cloud Computing services have met their targets, and understand any issues and implications based on the interpretation of their data analysis. Forth, CBP can be used as an independent way to evaluate any Cloud Computing businesses, projects and services, whether they are technical, or financial or user focused. Recommended ways to define the metrics, data collection, method of data analysis and interpretation of data analysis are essential to the development of Cloud Computing manufacturing. To present, measure and demonstrate the Cloud Business Performance becomes a priority for manufacturing organizations that adopt Cloud Computing. To highlight the significance of the effective use of CBP, the breakdown of this paper is as follows. Section 2 describes the related work for CBP. Section 3 describes a list of selected models and justification of the choice of the models. Section 4 presents the first case study for a small and medium manufacturing enterprise (SMME) that has adopted backup services for all manufacturing transactions and records. Section 5 shows the second case study for a software manufacturing organization’s forecasting on their business performance and risk. Section 6 sums up for discussion. Section 7 presents conclusion and future work for this paper.

2 Related work

This section presents the literature review and the related work to the use of Cloud Business Performance (CBP). Before discussing CBP in details, related work such as business performance measure (BPM), risk and return literature and the criteria for the CBP model will be presented.

2.1 Business performance measurement

Bouwman (2003) presents a detailed review about business performance measurement (BPM) and he focuses on defining the components and relationships for business models. He presents related literature for both computing and business research, and explains the essential components within the business models, and illustrates how the interactions within those business model components can contribute to BPM.

Figure 1 shows Business model components for business performance measurement, whereby Bouwman (2003) identifies four main factors and explains the relationships between each main component. He explains each component as the follows:

  • Customer Value of Service relates to customer satisfaction and evaluation. He identifies key factors to capture customer values and perception of service quality.

  • Organizational arrangements are about resources and capabilities that organizations can offer.

  • Technical arrangements are the IT functionalities and data, which include embedded processes, business processes and web services.

  • Financial arrangements include investment decisions and revenue models and each has its own focuses.

Fig. 1
figure 1

Business model components for business performance measurement (Bouwman 2003)

Full size image

According to Bouwman (2003), Customer Value of Service defines organizational arrangements, which need to transform to technical arrangements. The end products of technical arrangements enable customer value, and good customer experience means organizational goals are delivered. The success of technical work can generate costs but also produce revenues. Financial arrangements can determine the outcome of a new product or service, which often drives stakeholders to think whether their adoption is useful. Although Bouwman (2003) uses a number of papers to support his rationale, using financial performance is not the only way to determine the business performance (Samson & Terziovski 1999; Khajeh-Hosseini et al. 2011, Khajeh-Hosseini et al. 2012). There are other aspects such as customer satisfaction and technical efficiency which should be investigated. Bouwman (2003) explains performance indicators for organizations and summarizes selected literature. He recommends a performance dashboard introduced by Rayport and Jaworski (2001), who summarize the following metrics for business performance measurement:

  • Business model measures: These include the unique value proposition, capabilities and resources, exclusive partnerships and investment in technology.

  • Measures for branding and implementation: These include indicators for system up time, number of IT staff and the percentage of inaccurate orders.

  • Measures for customer acquisition: These include customer share, purchases and service requests.

  • Financial measures: These include revenues, profits, earnings per share and debt to equity ratio.

Suggestions from Bouwman (2003) and Rayport and Jaworski (2001) are important for organizations but neither includes details about how to measure and the systematic processes involvement in business performance measurement (BPM). This means BPM for organizations that adopt Cloud should be investigated further. However, this approach is generic and lacks the details about how to undertake quantitative measurement and data analysis.

2.2 Cloud business performance (CBP) overview

Cloud Business Performance (CBP) calculation is a strategic goal for organizations to enhance any values added and then minimize the risks of Cloud adoption (Dillon et al. 2010; Khajeh-Hosseini et al. 2012). Cokins (2009) explains relationship between business performance and management in terms of methodologies, risks and analytics, similar to the proposal by Papazafeiropoulou and Spanaki (2016) to balance between governance, risk and compliance. Business performance is related to organizational sustainability, which determines the survival and growth of organization. It consists of added values (such as profits or cost-saving) and risks of Cloud adoption. According to Cokins (2009), this is applicable to many sectors including emerging technologies.

Since benefits of Cloud manufacturing adoption include technical, financial and organizational aspects, CBP calculation requires including these three key elements. To demonstrate how CBP calculation can be performed in systematic and coherent ways, a framework should be proposed to help organizations determining the extent of success, case studies and any added values for Cloud adoption. For manufacturing organizations that decide to adopt Cloud, it will be useful to provide details for their CBP and service portability. Additional rationale to measure CBP is presented and each aspect of technical, costs and users considerations are explained as follows.

2.2.1 Costs (financial) measurement for cloud business performance

The challenge identified in this paper is a need to measure the performance of cloud in terms of its business benefits to aid the strategic decision of cloud adoption. This will address the key financial risk that needs to be addressed when adopting cloud. Similarly, the requirement to address the risk of the actual costs being different from the estimated costs, this can be caused by inaccurate resource estimates, changing prices or inferior performance resulting in more resources spent than expected. This is mitigated by monitoring existing resource usage and using estimation tools to obtain accurate cost estimates of deploying IT systems on the cloud. (Aubert et al. 2005; Khajeh-Hosseini et al. 2012; Dillon et al. 2010). The type of CBP measurement is focused on

  • Cost-saving: Inaccurate resource estimates can be reduced due to precise cost calculations and also consolidated resources to reduce operational costs.

  • Profitability: Calculates precise estimated and actual profits so that stakeholders can understand profits due to Cloud adoption.

2.2.2 Technical measurement for cloud business performance

There are requirements to ask whether Cloud adoption can provide better performance such as completing requests more quickly or whether more work can be done in the same period of time (Miller 2008; Armbrust et al. 2010). This relates to efficiency, which plays a crucial role in manufacturing since more outputs of new services can be produced than using the old services at the same time (Castellacci 2008). This type of CBP measurement is focused on:

  • Improvement in efficiency: the same number of jobs/requests can be completed quicker, or more jobs/requests can be done in the same time frame for Cloud manufacturing systems comparing to non-Cloud manufacturing systems.

2.2.3 Users (or organizations) measurement for cloud business performance

Khajeh-Hosseini et al. (2010) and Buyya et al. (2010) present organizational issues for adoption challenges which include whether the internal feedback is positive and the extent of user satisfaction rating. This is a measurement to reflect users and clients’ rating about Cloud manufacturing adoption, which is an important aspect to confirm the added values of using a new Cloud service. The type of CBP measurement is focused on:

  • Improvement in user satisfaction rating: An increased percentage of users (or clients) feel there is an improvement to the quality of products and services such as having a quicker response time, a higher proportion of jobs completed at the same time and a more efficient system/application to get their work completed, which results in a higher positive rating for Cloud manufacturing adoption. In general, this is summed up as user satisfaction rating.

2.3 Why CBP is important and the relation to organizational sustainability

While business performance measurement has been presented in Section 2.1 and 2.2, discussions about business performance should also focus on quantitative aspects, which can provide a more comprehensive review of Cloud Business Performance (CBP). Generic business performance should be reviewed prior to discussing CBP: how they are applicable to Cloud Computing and manufacturing organizations that adopt Cloud can then be discussed.

Organizations that adopt new technologies can experience change management, strategic management and IT management issues (Barras 1990; Grant 2010). This is the same for organizations that adopt Cloud (Grant 2010). Barras (1990) describes that a consequence of adoption of new technology or a new service may be a significant difference to change management and strategic management such as the way organizations go forward. For some organizations, it may mean a complete change to the way they deal with work, which can be processes, different skills, or business orientation. For example, technical staff need to spend more time with customer user support and training users how to use new systems. Business analysts can get their work done on the central private cloud and reduce the time for product analysis and development, and can spend more time with other tasks such as customer relationship management (Khajeh-Hosseini et al. 2011, Khajeh-Hosseini et al. 2012). Key metrics are undertaken so that stakeholders can understand business performance of their Cloud adoption. The above examples demonstrate the followings:

  • Changes in manufacturing organizations have impacts on organizational development from establishment, growth, saturation and decline phase.

  • New Cloud manufacturing adoption affects the way the organization operates.

  • Cloud manufacturing adoption fits well for business models, BPM and CBP to meet strategic focus for Cloud adoption.

Organizational Sustainability is the term which includes organizational development from establishment, growth, saturation and decline phase (Grant 2010). This can be applied to many types of Cloud-adopting manufacturing organizations. For example, a new start-up can focus on establishment of its new businesses strategies, products and services. For a large manufacturing organization that has adopted Cloud, it can be used as a pilot study and then focus on growth and its impacts to internal users and customers, or its added values to the business. Emphasis for organizational sustainability moves to how to sustain projects financially and how to extend the lifespan of technical projects. This has no exceptions to Cloud manufacturing that new strategies, product development and services should always be up-to-date and stay connected with the market trends (Kumar and Reinartz 2012). There are supporting cases in our proposed CBP method as follows:

  • A manufacturing organization can focus on the improvement in efficiency. For example, how much time can be saved to backup thousands of data across different clusters and sites comparing the private cloud storage and traditional backup services.

  • A manufacturing organization can focus on the rate of cost-saving or profitability that a new Cloud product or service can offer.

  • A manufacturing organization can focus on the improvement of customer satisfaction before and after Cloud adoption.

Those examples require accurate and systematic CBP for Cloud-adopting manufacturing organizations and metrics are useful to determine the extent of organizational development. In other words, collected data for CBP can be useful to determine the extent of return and risk associated to Organizational Sustainability. This allows stakeholders to understand whether Cloud manufacturing adoption meets their expected targets and finds out differences between their actual results and expected targets. Stakeholders can learn about the status of risk associated to Cloud manufacturing adoption and then identify ways for improvement to reduce risk. All these information is useful to support justification and added values.

2.4 Comparison with approaches that contribute to cloud business performance

This section investigates similar approaches that can contribute to the development of Cloud Business Performance (CBP). Li et al. (2015) demonstrate their intelligent approach to extract data and identify task for process mining, as an alternative for measuring business performance. However, it is not for Cloud-based approach which can be adopted by manufacturing firms. Weinhardt et al. (2009) present the early version of Cloud business performance with their conceptual framework and key performance indicators (KPIs). However, there is no any quantitative methods to measure CBP. Similarly, Garrison et al. (2012) describe success factors for organizations that adopt Cloud Computing, including manufacturing firms. However, they do not have any recommendations for quantitative measurement. Furht and Escalante (2010) have proposed the success factors and KPIs for measuring CBP. However, the KPIs have very simple mathematics and are focused much on costs. KPIs should also involve with the overall business performance that take into considerations for risk and return. Fundamentally, all businesses should take on risk and return seriously and regularly, and need emerging services and business analytics to understand the status of risk and return, so that businesses can make better decisions and be more competitive (Bughin et al. 2010; Demirkan and Spohrer 2016; Chang 2017a). This is particularly important for Cloud Computing manufacturing adoption (CCMA), which require regular revision of risk and return on manufacturing services. In this paper, the past and current CBP is adopted by Organizational Sustainability Modeling is focused on analyzing the past and present performance. The integrated Monte Carlo simulations and Black Scholes Model cam be used to predict the future performance, as illustrated between Section 3 and 5.

3 Methods for measuring cloud business performance

A successful Cloud adoption for a manufacturing organization is dependent on the management of both risks and benefits. A recommended way to manage risks and benefits is by the use of Cloud Business Performance (CBP), which requires a systematic method to collect metrics and calculate the associated risks and benefits. Different models are reviewed and compared.

3.1 Criteria and choices of methods for a cloud business performance (CBP) model

This section describes what the selected models are and how they can be useful for the Cloud Business Performance (CBP) model. Cloud adoption involves both strategic and operational activities. Khajeh-Hosseini et al. (2011) explain the link between strategy (management) and operations (staff) and their role in Cloud adoption. The role of staff on the operational activities is to ensure their work can be done effectively. But management likes to determine quantitative Cloud Business Performance (CBP), as “work done” does not indicate whether it meets management’s requirements (Kagermann et al. 2011; Khajeh-Hosseini et al. 2011). Skilton (2010) explain the importance of success factors for Cloud adoption. This makes identifying success factors important. Success factors need to work for both IT strategy and operations. The rationale is that the types of problems and challenges can sometimes be perceived as different by strategic and operational staff (Isom and Holley 2012). To address this, Hosono et al. (Hosono et al. 2009, Hosono et al. 2010) assert that there is a set of six core elements for Cloud and IT project management defining how important quality factors fit into their Non-Functional Requirements (NFR) for IT adoption. According to Hosono et al. (Hosono et al. 2009, Hosono et al. 2010) and industrial expert interviews conducted in our previous research (Chang 2014b), the core elements for computing project management are usability, performance, security, reliability, portability, scalability and industrial norms.

Additional explanation for each selected element is described as follows. Usability is important. Any Cloud services should provide easy to use features and users without prior knowledge should be able to use them easily without going for training or having any advanced knowledge. Performance is required because Cloud business performance must be fast, and able to complete tasks or provide services in an acceptable manner. Security is a core element because it ensures users are protected from malicious attacks and guaranteed that their Cloud platform or service is highly safe. Computational accuracy is crucial as cloud service is always available and stable, and calculations are always as accurate as possible. Portability is essential, as Cloud Services should be transparent so that users are able to use any application and platform from their desktops to Clouds without being concerned about the complexity of technologies. Scalability is a main characteristic for Cloud and users expect it as part of benefits for Cloud adoption.

3.2 Review of the suitable models for CBP

This section presents models which are useful to compute CBP. These models will use eight elements to review its suitability and compare with one another. There are not many quantitative modeling approaches for CBP. There are Mathematical models and specifically Monte Carlo, ARIMA, Black Scholes, CAPM and Organizational Sustainability Modeling (OSM) can be reviewed for use in CBP. Additional requirements for ‘industrial norms’ include the following:

  • Organizational data can be used to model – revenues, technical efficiency and growth for a manufacturing organization that adopts the Cloud.

  • The Cloud-adopting manufacturing organization’s growth and organizational sustainability;

  • A model should have flexibility to define the organizational focus which can be analyzed from collected data and then used for modeling.

A list of models is reviewed and presented as follows. All of them need the third party tools to improve its security, which is not stated again in each description.

3.2.1 Monte Carlo simulations (MCS)

Monte Carlo Simulations (MCS) is commonly used in financial modeling and analysis, and is useful to provide stakeholders about business performance and forecasting for their investments (Longstaff and Schwartz 2001; Moreno and Navas 2001; Choudhury et al. 2008; Hull 2012). It is optional to use data for MCS, and mathematical modeling in the banks does not use data. MCS is useful to predict an organizational business performance including their investment options, prices and risks related to investment (Choudhury et al. 2008). MCS can be used as an application to calculate prices and risks (Longstaff and Schwartz 2001; Moreno and Navas 2001; Choudhury et al. 2008; Hull 2012).

MCS is a complex model but due to the widespread use in many institutions and effort to make it user-friendly, it has a good usability. MCS is well-known for its performance and computational accuracy, and MCS applications can be used on different platforms. MCS is popular in industry and can be scaled for different environments (Longstaff and Schwartz 2001; Moreno and Navas 2001; Choudhury et al. 2008).

3.2.2 Black Scholes model (BSM)

Black Scholes Model (BSM) is used for financial risk modeling, particularly for stock market analysis (Hull 2012; Millers 2011). Millers (2011) demonstrates that BSM can be very effective to calculate the volatility, which is an uncontrolled risk that can affect business performance. BSM does not require any data. Although BSM performs well and offers good accuracy, it is a complex model and is not easy to use. Millers (2011) does not provide details for how BSM can be made scalable and portable, but this is possible with additional work. There is industry adoption but sophisticated software is often used and is not easily available (Hull 2012).

3.2.3 Organizational sustainability modeling (OSM)

Organizational Sustainability Modeling (OSM) is the improvement of the CAPM model that fully addresses the weaknesses of CAPM. First, OSM can process thousands of datasets at once and allows computational tasks to be completed in seconds. Second, CAPM is a generic solution for business performance measurement but not designed for Cloud Computing. OSM is designed to assess risk and return analysis for organizations adopting large computer systems, such as Cloud adoption for manufacturing organizations. Details about OSM and the supporting case studies have been demonstrated in Chang et al. (2016). OSM is a more suitable option than CAPM.

3.2.4 The choice of using which models for CBP

The choice of using which models to calculate CBP depends on the available data, strategies and types of focus of the manufacturing organizations. The first case study includes a small and medium manufacturing enterprise (SMME) that has significant collection of the previous business performance and has special interests to analyze and review their business before setting goals for their future targets. OSM is relevant since it is suitable to provide more comprehensive data analysis about the adoption of a new Cloud-manufacturing service. The second case study includes a new software manufacturing firm that has a large volume of cash flow including sales and expenditure. Their focus is to forecast their business performance and model large scale simulations on the worse case scenarios due to the volatile environment and challenging competitions involved. Thus, the integrated use of Monte Carlo and Black Scholes simulations can provide a more comprehensive forecast on their software manufacturing strategies.

3.3 The model for CBP for case study one: organizational sustainability modeling (OSM)

CAPM was developed to analyze the status of risk and return. However due to its limitations for being not able to handle large datasets and is not intentionally designed to analyze the status of risk and return for Cloud adoption, OSM is thus the preferred model for measuring and analyzing CBP. ARIMA and Black Scholes use mathematical formulation for forecasting and performance analysis (Saikkonen and Luukkonen 1993), which is not suitable for manufacturing organizations that emphasize on reviewing their past and quarterly business performance.

More complex rules and algorithms are required to be programmed in order to take in datasets and process data analysis. Monte Carlo simulations can score fairly well in all the shortlisting criteria except large datasets since additional codes need to be written to make it happen. Monte Carlo simulations are not designed to analyze the status of risk and return for Cloud adoption without significant efforts to make it happen, although such approaches can be adapted. For example, changing algorithms and focus from finance to Cloud adoption can be demonstrated by Chang et al. (2011) with significant efforts to change the system design, algorithm and large scale experiments.

The objective of OSM is to provide a systematic approach to help managers understand the status of risk and return of a project. The OSM formula is based on the original CAPM formula (3):

$$ e={\mathrm{r}}_c+\beta \left(a-{\mathrm{r}}_c\right) $$
(1)

where:

a is the actual return (or CBP) of a Cloud project, which stands for an actual CBP value.

e is the expected return (or CBP) of a Cloud project, which stands for an expected CBP value.

rc is the risk-control rate, the rate of manageable risk.

β is the beta value which represents a measure of uncontrolled risk.

OSM divides risks associated with an investment into two categories: those which can be controlled and managed, and those which cannot. Beta represents a value for uncontrolled risk. Thus, finding its values on each project is crucial. An approach for calculating beta is to perform linear regression, where the gradient of the slope is the value for beta (Sharpe 1964, 1992; Chang 2014a). Beta can be calculated by rearranging eq. 1, giving

$$ \beta =\frac{\mathrm{e}\hbox{-} {\mathrm{r}}_c}{\mathrm{a}\hbox{-} {\mathrm{r}}_c} $$
(2)

Risk-control has to be kept at 5% and below, since OSM treats risk analysis seriously. Since thousands of datasets are required, OSM requires a high quality of dataset and analysis. If risk-controlled rate cannot be sufficiently managed, it can influence and contribute to the cause of the uncontrolled risk. OSM can be used in three areas: technical, cost and user (or client) satisfaction before and after deploying the new solution. Suitable metrics should be defined for each. For example:

  • Technical: Efficiency gains may be measured by consideration of the completion times for Cloud and non-Cloud systems times. The risk-control rate is given by the proportion of failed requests/tasks.

  • Cost: Data will need to be collected regarding costs or profitability improvements arising from introducing new technology and the risk-control rate is the rate of gain assured even when targets are not met.

  • Users: Quality of service improvements which are usually evaluated from periodic user surveys. The risk-control rate reflects the rate at which incident happen.

Some details have been presented in Chang (2015), including the outputs, data processing, performance experiments, case studies and outputs by case studies. The focus for this paper is to strengthen on how CBP can be illustrated by the use of OSM model and emphasize that the outputs and data analysis can be useful for stakeholders. Different case studies are used to demonstrate the effectiveness of OSM and explain how it can make contributions to CBP research.

3.4 The model for CBP for case study two: The integrated uses of Monte Carlo and Black Scholes simulations

There is a second case study for a software manufacturing firm specializing in making software but use the resources built in the Cloud to enable software manufacturing. This can be achieved in a few varieties. First, make APIs and libraries in the Cloud and make them easily usable and be able to integrate with other APIs or libraries. Second, make those APIs and libraries plug-and-play and allow easy to build features. Third, automate the process so that the computer programs can make them. Forth, make the intelligent program to produce any number of outputs or outputs that allow simple forms of modification without involved in writing codes. Software manufacturing can also be used to produce 3D printing (Campbell et al. 2011; Berman 2012) and 3D virtual world (Bououd et al. 2016). The prototype can be designed by software and then be placed in the Cloud. Once confirmed by the user, a request can be sent from the Cloud to the 3D printing, so that outputs can be printed off. However, this business model is either not mature enough or competitive depending on which countries that services are located. The focus of this paper is to help those organizations predict their business performance and compute the detailed return (prices) and risk involved. The integrated use of Monte Carlo and Black Scholes simulations can achieve comprehensive analysis (Glasserman 2003; Asmussen and Glynn 2007).

4 Case study one: A small and medium manufacturing enterprise that has adopted backup services for all manufacturing transactions and records

This case study illustrates a small and medium manufacturing enterprise (SMME) that has adopted a Cloud Computing service that provides back up services for their own transactions, product development, trading, business and client records. The intention is to blend the enterprise resource planning together with Cloud manufacturing. The integrated Cloud service allows this SMME to stay competitive since the efficiency can be improved. This SMME has adopted the methodology for Cloud services recommended by one of our research work (Chang and Wills 2016), whereby the methods of backing up biomedical data have been analyzed and presented in details. Transferable lessons can be achieved. The focus is to “replicate the success” recommended by the previous research contributions, which demonstrate how to design and deploy Cloud backup services that allow thousands of files to be backed up efficiently. Even if there are reported problems, backup process can be continued to ensure that the completion of the task can take the priority. The use of Organizational Sustainability Modeling (OSM) can analyze the status of return and risk incurred as a result of using both Cloud and non-Cloud backup services. Thus, the lesson learned in the previous study can be adapted by this SMME to measure the key metrics used by OSM for analysis. Similarly, when all the automated backup is fully functional, it can be regarded as an automated manufacturing backup system for the enterprise.

4.1 The OSM metrics and measurement

This section presents the metrics required by Organizational Sustainability Modeling (OSM). Beta represents the uncontrolled risk. If the actual values are higher than the expected values, the beta is less than 1. This can be interpreted as showing that the uncontrolled risk is low and that the Cloud project is not exposed to a high level of volatility. In this case study, OSM is focused on the technical adoption of Cloud Computing, with a focus to investigate improvement in efficiency, which refer to the rate of time difference before and after using Cloud manufacturing backup and enterprise service. To demonstrate the use of OSM, three key metrics required by OSM are as follows.

  • Actual completion time: This is the actual time taken to complete the backup process while keeping the rate of failed commands under 5% to ensure controlled risks can be fully managed. Another round of re-running failed commands will be carried out.

  • Expected completion time: This is the expected time under ideal conditions where there will be no failed commands. It can be calculated by the network simulator prior to using the subsystem.

  • Risk-control rate: This is the controlled risk of running comparisons. The rate of failed commands should be kept under 5%. If the rate is less than 5% then failures will be reported but will not interrupt the backup process. However, when the rate rises above 5% the entire backup process is terminated and will be restarted on both systems.

A list of data variables, their definitions and explanations is presented in Table 1.

Table 1 Overview of the SMME Case Study: Metrics 1
Full size table

4.2 Key OSM results and analysis

The OSM outputs have been explained in Chang et al. (2015, 2016), which include beta, standard error, Durbin-Watson test, mean square error and R squared values with all their interpretations. In this section, results and their interpretations are presented in Table 2.

Table 2 OSM for the SME on analyzing improvements in efficiency
Full size table

  • Beta is equal to 0.91543. This is a medium-high value although it is close to but below 1. The project itself has a medium-high volatility of uncontrolled risks. It may mean the current services will reach its bottleneck. Either more upgrades on existing hardware or provision of extra services are required to ensure that services can cope with increasing demands in back up services.

  • Standard error is 0.10607 and is relatively low, which suggests there is extremely high consistency between all metrics with very few outliers.

  • The first order Durbin-Watson result is above 1 (Durbin et al. 1950, Durbin and Watson 1951), and Pr > DW shows that there is a high negative autocorrelation (0.9994) favoring OSM, which means a good quality of data and standard errors. The positive p-value is 0.0006 and is acceptable.

In addition:

  • Mean Square Error (MSE) is 5.85622, which suggests a wide variation between three different groups of users. The first group is the majority of around 60%, which has a fair expected and actual rate of improvement (between 5% and 10%). The second group consists of around 30% of the sample population and has a wide positive difference in expected and actual rate of improvement, which has 10% and 20% expected and actual rate of improvement. The third group consists of about 10% of the sample population and they have a higher expected rate of improvement than the actual rate, although both rates are positive. It is important to find out any reasons behind their scores and interviews with users should be undertaken before their next survey.

  • Regression R-square is 0.6053 and it is optional to use 95% C.I to analyze. Regression with 95% C.I is 0.6502, which means all data points can be fitted in the regression. R-squared values can determine the source of risks are external or internal (Teoh and Zhang 2009). This means 60.35% risks are from external due to the increasing demands and competitions. 39.95% of risks are from internal such as lack of system upgrade or purchase of new software.

  • Additional tests include the t-value test and F-value test. The higher their values with low Pr > |t| and Pr > F, the more accurate the outputs are towards the expected values set by the hypotheses. The high values of these two tests suggest the results of regression and analysis are fairly consistent and accurate.

4.3 Analyzing cloud business performance via the datapoints

All the OSM datapoints can be used to analyzed for regression and check whether they are all fairly consistent with all datapoints suggested by the summary of statistical analysis in Section 4.2. With regard to this, beta is the uncontrolled risk value that can influence the way that a Cloud manufacturing business or a Cloud manufacturing service to move forward. If a Cloud business or service is rated as volatile, more actions and contingency plans should be in place to ensure that the Cloud business or project can be delivered more smoothly. The input variables corresponds to the actual completion time, expected completion time and risk-control rate, the latter of which is defined as the rate of failed jobs that can be controlled and will not affect the overall quality and delivery of Cloud manufacturing backup and enterprise service. The input variables have been collected over a period of three years and have been fed into OSM for regression. Outputs have been presented in Fig. 2 below, whereby three regression lines have been computed based on the datapoints. The blue line in the middle is the regression line to measure beta and the other two lines correspond to the upper and lower 95% confidence interval lines. OSM calculations confirm that beta is 0.91543.

Fig. 2
figure 2

The linear regression to calculate beta values

Full size image

OSM for Cloud Business Performance (CBP) service can also be directly computed on this SMME’s Cloud manufacturing service, which can save their data analyst significant effort to compute. Data can be directly computed in the Cloud. Outputs can be presented in both numerical and analytics formats, the latter of which is shown in Fig. 3. Results from automated service are equally consistent with services manually performed by analysts. Due to the agreement, further details on the central part of the core technologies cannot be released.

Fig. 3
figure 3

The OSM CBP service directly computed on the SMME’s Cloud manufacturing service

Full size image

In the OSM CBP service, data visualization is a significant part to demonstrate the outputs and further exploit the implications from the data analysis. As shown in Fig. 4, data visualization allows analysts to study any implicit interpretation from analysis. For example, the statistical residual of an observed value is the difference between the observed value and the expected value. The smaller the value, the more accurate the result. When comparing residual with statistical analysis, the data behavior follows the normal distribution. Cook’s distance measures the extent of influence of data points, including the effect of deleting a given observation. It should be lower than the recommended benchmark, which shows one datapoint is not. The variable “rate_actual” compares the outputs of improvement in efficiency (the difference between using the new and old services) and finds that the results are fairly consistent.

Fig. 4
figure 4

Data visualization of the OSM CBP service

Full size image

4.4 3D visualization to present business performance

3D Visualization is a crucial aspect of OSM to identify any unexploited area for data analysis and understand the overall trend of the business performance based on the collected data. In a previous work, Chang et al. (2016b) demonstrate the usefulness of 3D Visualization to confirm the improvement of user satisfaction rate for learning and help the stakeholders to understand the interpretation of data analysis. Lesson learned can be applied for this case. All the datapoints can be computed in 3D Visualization with an overall upward direction shown in Fig. 5. The x-axis is the actual rate of business performance, y-axis is the expected rate of business performance and the z-axis represents the risk-control rate. In this case, business performance is focused on the improvement in completion time, whereby the expected rate and then actual rate have been recorded and compared by following steps in Chang and Wills (2016). Risk-control rate means the percentage of failures or incomplete tasks out of all the measurement. Outputs allow the interpretation of complex CBP more easily without the stakeholders using mathematical algorithms and analysis. Fig. 6 shows the 3D Visualization in 90 degrees to identify any missing areas left for interpretations. Despite there is a general upward movement, the Cloud manufacturing service has undergone periods of ups and downs hidden underlying the 3D Visualization. It can be interpreted that ups and downs happened in the Cloud manufacturing service had been managed under control which could contribute to positive outcomes with a progressive improvement.

Fig. 5
figure 5

3D Visualization for business performance of the first case study

Full size image
Fig. 6
figure 6

3D Visualization for business performance of the first case study in 90 degrees

Full size image

5 Case study two: a software manufacturing firm forecasting her business performance

As discussed in Section 3.4, software manufacturing brings an innovative business concept to print software-based products as 3D items. However, there are challenges ahead since a huge investment is required and there is an uncertain period going through over competitions ad market volatility (Turban et al. 2015). The emphasis for CBP here is to forecast business performance including the return and risk analysis. In terms of return, the ideal call and put prices to make and the extent of risks involved in investments and operations. Results can provide investors and stakeholders useful details for their decision-making. The integrated Monte Carlo and Black Scholes simulations can be used to provide a more thorough and comprehensive analysis.

Monte Carlo Simulations (MCS) has been widely used to compute the risk and prices for different types of investments through large-scale simulations (Choudhury et al. 2008; Jahangirian et al. 2010). Black-Scholes-Merton (BSM) model can be used to simulate market complexity, volatility and risk/return ratios and have track record in supply chain finance (Hull 2012; Millers 2011 ). MCS can be integrated with BSM model to perform full and comprehensive high frequency simulations (Chang 2014b) so that all pricing options can be calculated under all possible scenarios and circumstances. The combined effort can ensure a more reliable and accurate approach than using one model. Visualization and analytics can ensure that investors and stakeholders without finance or computing background can understand the interpretations of results better than dealing with numerical computations directly. The algorithms based on the MCS and BSM can be written and the input variables are based on the market demands, investors’ targets and the trading decisions. Based on all the inputs, the integrated MCS and BSM model can perform all the stress tests and present visualization and analytics as outputs.

5.1 Manufacturing risk calculation by Monte Carlo simulations

There are different types of risks exposed to manufacturing, including price fluctuation, demand and supply management, availability of raw materials, clients’ financial status, global and local economy that the manufacturing organizations are based (Kouvelis et al. 2006; Manuj and Mentzer 2008; Jahangirian et al. 2010). There are complex factors determining the trends, prices and market directions. Even proposals from Kouvelis et al. (2006) and Manuj and Mentzer (2008) about the global supply chain management cannot eliminate risks and identify all of them. Additional methods will be required. Monte Carlo simulations (MCS) have been very useful to provide accurate and high-performance calculations for supply chain management to ensure that all the risks can be carefully calculated on regular basis (Hull 2012; Mascarenhas et al. 2015). Results can provide decision-makers better alternatives, more accurate and more realistic comparisons to the market trends. Least Square method (LSM) is a precision method in MCS used to calculate results to be as accurate as possible. Risk calculations can be represented by giving it a price. If the risk price is rated as 24.0%, or 24 out of 100, then it means that the probability of such a event to happen is 24.0%; or the occurrence of risk will cost 24 unit of price out of 100 as a loss as demonstrated by Chang et al. (2012). LSM can be adopted to run 100,000 simulations in the cloud within a few seconds. There are two types of risks, the American option to calculate the price at any time (in this case, it means ‘now’) and the European option.

The following is the result of running LSM to calculate the expected risk price.

  • MCAmericanPrice = 1.435 (risk price)

  • MCEuropeanPrice = 1.387 (risk price)

This means the average performance for risk price is 1.435 (1.435% for operational risk to happen). The best risk pricing that the completion of project or the end of investment (exit/expiry) to happen is 1.387% (1.387% for operational risk to happen). Similarly, the 95% confidence intervals can be provided in the simulation to ensure that upper and lower range of the likely risk prices can be covered. Table 3 shows that risk prices if using both European and American options. Results can help decision-makers to know the extent of risk and differences between results computed by two methods, so that they can devise the suitable strategies for their strategies and operations.

Table 3 Calculation of the risk prices for European and American options
Full size table

Manufacturing products can be subject to market fluctuations such as currencies, supply-demand, economies of the client countries and so on (Kouvelis et al. 2006; Manuj and Mentzer 2008). A sophisticated approach is to calculate European and American options for risk prices for at least monthly basis, so that any factors that can affect the values of the products can be monitored and reviewed. This has no exception to software manufacturing products ranging from games, software for medicine, software for 3D printing and software for car manufacturing. Since the identity of the case study firm cannot be revealed, the information that can be shared is that their product sales are dependent on the seasons. For example, the March–April, June–August and December are the hot seasons that they have the lowest risk values due to higher demands in the products and there are off-peak seasons that low sale records may lead to the change in their forecast business performance. Fig. 7 shows risk value/price with American and European options in Year 2015. Risk values in American option are always slightly higher than the European option. The main reason is that European option deals with improving return and managing risks at the time of the month, in which the firm has a better knowledge on the strategies, products and services they can control. While taking medium-term of risks into considerations, there is a higher risk that may include uncertainties in currencies, supply-demand and economy of the client countries before the end of the following financial year. Results in Fig. 7 can be useful for decision-makers to focus more on the hot seasons they do well, such as scaling up their businesses and operations and scaling down in off-peak seasons to reduce operational costs.

Fig. 7
figure 7

2015 risk value/price with American/European options

Full size image

All these risk simulations can be scaled up to 600,000 simulations per service to ensure that large scale simulations do not miss any possibilities and scenarios that risk can happen, with other detailed results to be presented in Section 5.2. All the input variables include the call price, put price, market volatility based on the Chicago Index (Chang 2014b), date of expiry and any targeted inputs requested by the investors or decision-makers. The Cloud is a hybrid Cloud as implemented in the previous work (Chang and Wills 2016) which allows large scale simulations and experiments to be concurrently performed and executed. Execution time was recorded five times to get the mean values. Results of simulations are shown in Fig. 8.

Fig. 8
figure 8

Execution time for running up to 100,000 simulations for risk pricing

Full size image

5.2 Large scale simulations of asset prices by the integrated method of Monte Carlo simulations and Black Scholes Merton model

Monte Carlo simulations (MCS) can be integrated with the Black Scholes Merton (BSM) model to perform full and comprehensive high frequency simulations (Damodaran 2008; Chang 2014b) so that all pricing options can be calculated under all possible scenarios and circumstances. The resources to undertake and experiments have been described in Chang and Wills (2016), where large scale simulations can be performed in Data Center. As discussed in Section 5 earlier, the integrated MCS and BSM can perform large scale simulations. The interest is to simulate the range of asset prices which are subject to the impact of global supply chain and economy for manufacturers. All the input variables can be taken based on put price, call price, market volatility, maturity period (100 weeks) and key variables relevant for both MCS and BSM. Figure 9 shows all the asset price fluctuations of the manufacturing products/services offered by the software manufacturer. They have a starting price of $60 per unit price and can forecast that at the end of 100 weeks, their values are ranged between $56.40 and $66.0 per unit price. Results can provide the stakeholders valuable insights about their forecast values. If the management aims for the highest value of $66.0 per unit price, then they will have to closely monitor their business performance including sales, trading activities, supply-demand and popularity of their products to ensure that their products can be on the way up and minimize all the possibilities that can reduce their prices.

Fig. 9
figure 9

Simulated asset paths

Full size image

Figure 10 shows the execution time for running up to 600,000 simulations for asset paths simulations of Fig. 9. Five sets of simulations have been performed and recorded, the mean value is the execution time taken for each type of simulations, ranging between 100,000 and 600,000 simulations per service. The time taken for 600,000 simulations per service take 34.37 s. In the second case study, the software manufacturer can forecast their business performance and can complete large scale of 600,000 simulations win 34.37 s.

Fig. 10
figure 10

Execution time for running up to 600,000 simulations for asset paths

Full size image

6 Discussion

There are two case studies presented in this paper demonstrating a new concept of Cloud Business Performance (CBP). The first case study uses OSM to model and analyze CBP for the manufacturing organization. Results show that data used for analysis are highly consistent; the uncontrolled risk is medium-low and 60.53% risks are from the external risk. The use of 3D Visualization help the stakeholders to oversee the relationship between expected rate, actual rate and risk-control rate with regard to the adoption of their backup services for their manufacturing transactions and records. The second case study focuses on a software manufacturer that forecasts its business performance and risk. They use MCS and BSM to compute large scale simulations of risk price value and asset price values. While scaling up to 600,000 simulations per service, the execution time is under 35 s. In these two case studies, our research contributions include:

  1. 1.

    Present the new case for Cloud Business Performance (CBP).

  2. 2.

    Use suitable models to analyze and compute CBP for two different types of manufacturing business focus and justify the results with the analysis, paying particular attention to the status of return and risk.

  3. 3.

    Interpret the analysis and implications for the stakeholders and decision-makers to take appropriate actions. The first case study is focused on analyzing the “past performance” and the second case study is focused on predicting the “future performance”.

Existing literature does not have any CBP definitions, case studies, methods used and detailed results and analysis that have been illustrated in this paper. The new case study will develop forecasting methods based on the improvement of forecasting model presented by Chang 2017b), whereby “weather derivatives” have been developed to forecast temperatures in Sydney, Singapore and London between Year 2012 and 2014. ARIMA is the model behind forecasting and it can be further developed for CBP. Furthermore, advanced methods have been developed to present complex data in visualization within seconds, so that anyone can understand weather temperature outputs more easily. The next step of weather services also includes the measurement of accuracy between the predicted and actual results, and also the measurement by F-measure in all the expected and actual results. These valuable lessons can be transferrable to improve the quality of CBP measurement in analyzing the past performance and understanding any correlation between the past, present and future trends or results.

Additionally, this concept can also be applied in business intelligence as a service (Chang 2014b) to perform a large scale pricing and risk model in real time for selected stocks. To make an evolutionary approach, all services should collect a large quantity and size of data and can process, understand and analyze as quickly and accurately as possible. Recommendations from Internet of Things (IoT) (Whitmore et al. 2015) and Big Data (Hota et al. 2015) can be jointly incorporated and delivered. IoT is a platform to collect data and Big Data provides advanced algorithms to understand, analyze and recommend improvements for business performance.

7 Conclusion and future work

This paper presents a new concept, Cloud Business Performance (CBP) and describes the method of measurement, data analysis, impacts to manufacturing and case studies about CPB for manufacturing organizations. Three methods with two case studies have been demonstrated to validate CBP. The first case study presents a small and medium manufacturing enterprise that has adopted backup services for all manufacturing transactions and records. OSM is the model to analyze all the data and explain the implications of all the outputs, with regard to uncontrolled risk, external risk, the accuracy and validity of analysis and the comparison of three key metrics in 3D Visualization. Data visualization has been used to help the stakeholders spot any areas that are difficult for exploitation and interpretation for the business performance data. The second case study shows a software manufacturing organization’s forecasting on their business performance and risk. Both MCS and BSM have been jointly used to provide better quality of forecast for risk, including results from American and European options. Up to 600,000 simulations can be scaled up to ensure that all the scenarios of the asset price paths can be fully presented to allow the decision-makers to know the worth of their products after 100 weeks of time, so that they may plan for new products or new services for the existing products. Methods used, results and analysis have been fully justifiable to support the case of CBP for manufacturing organizations. We have demonstrated that the use of CBP calculation and prediction analysis is useful for manufacturing organizations that adopt Cloud Computing. Transferrable techniques from ARIMA, Internet of Things and Big Data can be jointly delivered to make the next generation of services for business intelligence, weather forecasting and service sectors.