Toward energy-efficient cloud computing: a survey of dynamic power management and heuristics-based optimization techniques

Abstract

Cloud computing is the most prominent computing paradigm in the present era of information technology. However, data centers needed for hosting cloud services demand huge amount of electrical energy and release harmful gases to the atmosphere. To ensure a sustainable future, there is a need to focus on energy efficiency in cloud computing. Early literature pertaining to energy consumption in cloud computing is primarily focused on individual sub-domains like scheduling techniques, optimization, and green computing metrics. Research literature on cloud resource optimization is found to be the most discussed but less structured. This paper intends to provide a complete picture of energy efficiency in cloud computing. It also classifies heuristics-based optimization methods and the dynamic power management techniques. The survey shows the research trends based on regions, journals, conferences, etc., in the domain of energy efficiency in cloud computing. The study concludes with research issues and future research directions.

1 Introduction

Energy consumption and greenhouse gas (GHG) emission by Information and Technology (IT) industry is increasing due to technological advances. It poses a severe threat to the environment. The threat is drastically increasing due to the increased use of computing in all aspects of life. According to Times report (2014), in the next half a decade, online users are expected to increase by 60% which will result in an increase in data consumption. It was predicted that data consumption would triple in 2012–2017, amounting to approximately 121 exabytes [1]. An IBM report “Ten key marketing trends in 2017” states that 2.5 exabytes of data are generated and consumed every day amounting to approximately 900 exabytes a year [2]. Internet users increased by 50% in 2018 [3]. As a consequence, there is an exponential increase in power requirement for data centers. This leads to severe environmental issues [4]. The demand for cooling infrastructure to control heat dissipation in data centers will also increase. Cooling infrastructure again would cause overhead by consuming more power and releasing GHGs. Maximum emission is due to electricity, air conditioning (AC), steam and gas supply [5]. According to a report of International Energy Outlook, during 2010–2040, the consumption of energy in the world will increase by 56%. Major consumers will be IT organizations [6]. The fact that the world will face an energy crisis is certain. So, it is a necessity to monitor energy consumption for a sustainable future. According to the World Wide Fund for Nature (WWF) report “A lack of access to energy is one of the main causes of poverty.” The world needs a drastic reduction in carbon dioxide (CO₂) emissions within the next few years to avoid catastrophic climate [7].

Cloud Computing (CC) offers a promising solution for energy efficiency as it provides a virtualized environment which leads to multi-tenancy. Tarandeep et al. [6] state that “Cloud installations have higher server utilization levels and infrastructure efficiencies. Due to improvements in utilization levels achieved without compromising the desired performance, the role of CC in achieving energy efficiency has gained researchers’ attention.” The continuous acceptance of CC will reduce energy consumption in the data center by 31% during 2010–2020 [8]. Efforts are made by various organizations (Facebook, Google, and Amazon) to build energy-efficient data centers, and the research community has already taken it as a challenge [9].

Energy efficiency in CC has its own research problems and challenges as defined in Sect. 8. This article focuses on scheduling as it serves as a generic solution by effectively mapping tasks to efficient resources. Scheduling enhances other energy-efficient optimization solutions as discussed in Sect. 2.2. Several researchers have focused on energy efficiency, and a lot of literature is available. Further, most of the literature is published in conferences due to the dynamic nature of this domain [10]. As per our knowledge, a survey article has not been framed which addresses energy-efficient scheduling through heuristic-based optimization techniques and provides metrics on green cloud computing (GCC). Integration of sub-domains (scheduling techniques, optimization, and green computing (GC) metrics) is required to provide a complete overview of energy efficiency. Beloglazov et al. [11] presented a pioneering survey on energy-aware data centers and CC systems. It provides a taxonomy of hardware techniques, but it has a limited focus on scheduling techniques. Yu et al. [12] presented scheduling algorithms for grid computing specifically. Wu et al. [13] presented workflow scheduling for CC. The article mainly focused on scheduling techniques in contrast to energy awareness. Kaur and Chana [6] presented a survey on techniques for achieving energy efficiency in CC, but it did not provide a classification of heuristics-based optimization algorithms. Another Pioneering survey by Jing et al. [14] focused on processor server storage and cooling infrastructures. So, there is a need for a precise and concise survey to build a background for further research. The WWF report concludes on increasing the research on energy awareness and production of renewable energy. It indicates that the current expenditure on research and development on energy efficiency is €65 billion a year globally, out of about € 900 billion total expenditure across the globe on research and development in all segments. This expenditure must be increased twice in the next decade [7].

This survey is framed to provide a clear view of energy efficiency in CC through scheduling-based optimization. This survey makes it clear that scheduling- or heuristics-based optimization algorithms are a key solution for achieving energy efficiency in CC as discussed in Sect. 2.2. The survey tries to deliver a concise knowledge base about GCC using heuristics-based optimization techniques. The article discusses various concepts of energy efficiency and classifies the literature according to dynamic power management and heuristics-based optimization techniques.

The survey consists of nine sections. It is framed in a sequential manner covering all aspects related to energy efficiency using a problem-solving approach. Other sections are framed in the following manner: Sect. 2 describes the background on GC, CC, and their integration. It explains various energy-aware optimizations and concludes that scheduling is a holistic solution. Section 3 defines research questions and the adopted methodology for the survey. Section 4 presents a detailed overview of scheduling, energy efficiency, and their interrelation following a W4 approach. Section 5 gives details about the solution (optimization). Classifications of dynamic power management and heuristics-based optimization methods are presented. Section 6 describes metrics to measure the effectiveness of the solution. Section 7 provides global analysis to discover recent research trends. Section 8 provides research issues and a strategy for future research. Finally, conclusive remarks are provided in Sect. 9.

2 Background and motivation

Energy consumption in the data center has got ample consideration recently, but still many issues have not been addressed yet [15]. These issues are described in Sect. 8. A sustainable future needs energy-efficient techniques and reduction in GHG emission by cloud data centers.

2.1 Cloud computing and green computing

The National Institute of Standards and Technology (NIST) defines CC as “a model that facilitates expedient and dynamic access to a large pool of computing resources which can be shared, dynamically allocated, and discharged without much managerial involvement or service provider interaction” [15]. Virtualization allows several Virtual Machines’ (VMs) generation on a single physical machine [16].

GC includes planning, developing, consuming, and organizing of computing services in an environmentally friendly approach to promote sustainability [17, 18]. CC along with GC can prove to be a boon by employing energy-efficient computing practices at various service models. Service models are Infrastructure as a Service (IaaS), Software as a Service (SaaS), and Platform as a Service (PaaS) [19].

Figure 1 illustrates the GCC framework. Thus, application software, platform, tools, and infrastructure must be energy-efficient to promote GCC practices. Details regarding GCC techniques will be discussed in Sect. 2.2.

Fig. 1

Green cloud computing framework

Cloud service providers (CSPs) deliver services to users by ensuring the security of data, quality of service (QoS), load balancing and traffic management as per the Service Level Agreement (SLA). Scheduling constitutes a generic solution for managing all these requirements [10]. Challenges in CC include (1) energy efficiency (2) QoS (3) SLA compliance (4) load balancing (5) security (6) traffic management and (7) cost-effectiveness.

These problems are addressed by using energy-efficient equipment, cost-effective provisioning, routing protocols, and encryption algorithms, etc. But scheduling can constitute a generic solution. Energy efficiency: Resources should be scheduled in such a manner that it minimizes carbon footprint by balancing the trade-off between overutilization and underutilization of resources.

Scheduling can be a solution in maintaining QoS-based application performance in the cloud. SLA compliance: CSPs have to complete the task on time and in the specified budget without affecting the reliability. So, optimized resource management requires scheduling based on makespan, deadline, and budget constraints. Load balancing requires mapping of VM resources to physical resources which involves scheduling algorithm. Security: Tasks are mapped to private or secure resources through scheduling. Traffic management: Data routing and forecasting techniques are in the solution domain, which involves scheduling for accessing cloud services efficiently. Cost-effectiveness: In this case, VM resources are scheduled in such a way that the total cost of deployment is reduced. Table 1 summarizes the literature to make it evident that scheduling is a generic solution for most of the issues in CC.

Table 1 Summary of research issues in cloud computing

CC has transformed the way computing services are delivered. Although it brought a revolution in our lives, it cannot be considered a disruptive technology alone as addressed by [34]. Resources in data centers consume 3% of global energy which is likely to get triple in the next decade. They are also responsible for 2% of GHG emission which will be 16% in the near future. It has been predicted in the year 2047 that there would be an enormous temperature on earth which leads to unsuitable living conditions [37]. So, CC may not be disruptive. But CC services availed in an energy-efficient manner can be disruptive. Major IT organizations have initiated efforts toward GCC. Apple as a leader has committed using renewable energy to power its iCloud. The organization deployed geothermal and solar energy to power its data centers. Google, Yahoo, and Facebook have started using renewable energy in their data centers as reported by Greenpeace [38].

2.2 Energy-efficient approaches for green cloud computing

There are three types of energy-efficient optimization techniques for cloud computing: (1) Infrastructure-based optimization which deals with infrastructural changes like making green buildings using energy-efficient equipment, air-conditioned racks, perforated tiles, floor raising, and other cooling equipment [39] for thermal management of data centers. Energy-efficient power distribution and energy star rating equipment [40] are infrastructure-based solutions, but are expensive to implement [41] and provide a limited reduction in energy consumption. Figure 2 represents infrastructure-based optimization in CC. (2) Hardware-based optimization: It includes employing multi-core architectures, voltage, frequency scaling, parallel architectures, energy-efficient hardware components, dynamic component deactivation, and consolidation. Angel et al. [42] were able to obtain significant energy consumption reduction using approximation algorithms on unrelated parallel machines. Dynamic voltage–frequency scaling (DVFS) practices are used on computing components for assisting the dynamic amendment of their performance uniformly to power consumption. Major techniques used are DVFS and DVFS with slack reclamation. DVFS and slack reclamation are described in detail in Sect. 5.1.

Fig. 2

Infrastructure-based optimization techniques

Figure 3 shows the architecture of a system using DVFS. A significant part of the research community is working on power management and scaling methods (Sect. 5.1). Liu [4] presents a renewable energy-based uninterrupted power supply scheme with an inbuilt central controller for power management. DVFS was used to tune the power demand of the server to get the efficient power point. Integer linear programming was also used. Results showed that the proposed framework improved backup energy capacity by 28%. Energy-efficient hardware components include efficient network and storage devices. A framework that implemented control strategies for the network and used network devices for local control mechanism was designed resulting in power savings [43]. Consolidation means to combine or integrate into one. Resources can be consolidated onto a lesser number of machines by halting underutilized machines to manage energy consumption. Consolidation can be performed at various levels: (a) VM consolidation, (b) server consolidation, and (c) task consolidation. Figure 4 represents the VM consolidation technique.

Fig. 3

Architecture of a system using DVFS

Fig. 4

VM consolidation technique

Consolidation results in optimization of hardware or infrastructure, but it involves the use of migration algorithms that come under software-based optimization. Further, hardware-based solutions are not sufficient unless resources are properly utilized. Utilization of resources depends upon scheduling. So, scheduling helps to optimize the infrastructure and hardware usage too. As per the literature, energy efficiency was not much achieved even after infrastructure or hardware changes due to poor design of software and programs. So, software-based optimizations are necessary. (3) Software-based optimizations include resource scheduling, allocation techniques, migration algorithms, throttling, and use of parallel programs. Scheduling is a holistic approach used everywhere. Being a non-polynomial (NP) Hard problem, scheduling can be done by various optimization techniques (as discussed in Sect. 5.2). The literature shows that scheduling algorithms have been widely used for minimizing energy consumption. Diaz et al. [44] utilized heuristics as a base of a scheduling algorithm to map the task on the heterogeneous system while reducing energy consumption. A model that used bi-level multi-objective programming for the locality and energy-aware multiple jobs scheduling was proposed, and numerical experiments indicated the effectiveness of algorithm [45] for reducing consumption of energy.

3 Research methodology

Table 2 illustrates the objectives and limitations of existing surveys. Surveys of highly reputed journals (SCI) with significant citations, most relevant subtopics are taken into consideration.

Table 2 Focus and limitations of existing surveys

A total of 879 articles were excluded by reviewers on the basis of relevance, 671 were excluded on the basis of research contribution, and 256 were excluded on the basis of abstracts. Finally, 103 articles were left that clearly focused on the objectives of the survey. 245 references of publications published after the year 2010 were identified, and 167 were eliminated due to redundancy. Some not relevant articles included data networks (39), microprocessors (29), grid computing (10), and other irrelevant topics. The remaining 78 were inspected by reviewers, and 46 were included according to inclusion criteria (Fig. 5). The total articles left after this procedure were 156. The inclusion criterion was related to energy-aware scheduling in CC through optimization, measuring performance through GC metrics and GCC. Classification criteria (dynamic power management and heuristics- or scheduling-based optimization techniques) are finalized, and the literature is classified as discussed in Sect. 5. Table 3 illustrates the sets of keywords used in the title of surveyed articles, and these articles are classified in Sect. 5. Figure 6 shows the percentage of articles included in the survey according to keywords occurring in the title. Table 4 lists research questions and motivation.

Fig. 5

Exclusion and inclusion criterion

Table 3 Sets of keywords in title of surveyed articles

Fig. 6

Frequency of articles surveyed according to keywords in title

Table 4 Research questions and motivation

4 Scheduling: a problem

Scheduling, in general, is a process of mapping of tasks to resources or target machines based on a criterion. Inefficient scheduling may result in performance degradation. According to the literature, scheduling is classified into three types: static, dynamic, and hybrid scheduling.

In static scheduling, the task execution environment and its characteristics are known in advance. Mapping of tasks to resources is determined before execution. At compilation time, information about cost and execution time is known. Jing Mei et al. developed an energy-aware scheduling algorithm by minimizing duplication and assuming task execution time, data size, and task dependencies are known before execution [52].

In dynamic scheduling, tasks are mapped to resources at runtime. Execution time and cost are available at runtime. A dynamic scheduling algorithm based on earliest deadline first (EDF) and power scaling method was used in hard real-time systems to reduce energy consumption [53]. Another dynamic scheduling algorithm used a multi-objective function to decrease the consumption of energy. The algorithm used resource allocation methods based on heuristics and was employed on parallel task-based applications. The results indicated significant energy savings for different scenarios. In directed acyclic graphs (DAG) types of embarrassingly parallel, matrix multiplication, and scatter–gather, the average energy savings were − 22.44%, − 33.17%, and − 31.50%, respectively [54].

Hybrid scheduling is a combination of both dynamic and static scheduling strategies. Cost and execution time can be predicted at compilation time, but tasks can be assigned to resources only at runtime. Thus, it is statically planned, but dynamically scheduled. An energy-efficient algorithm which helped in achieving energy efficiency in smartphones used both dynamic and static scheduling practices. Results indicated significant improvements in energy savings [55].

Other than these types, the term “workflow scheduling” is commonly found in the literature. Workflow is a paradigm that represents various applications which are computationally complex. It is automation of procedure to process data by following certain rules. It represents various applications such as big data processing, scientific applications, web applications, data analysis [13, 19]. Many workflow applications are migrated to CC because of its ability to signify an extensive range of activities. Types of workflows are (1) abstract, (2) concrete, (3) business, and (4) scientific workflows. Abstract workflows provide tasks in an abstract form without describing specific resources and providing flexibility to users. It gives only service semantic information. Concrete workflows describe tasks for specific resources and give both service semantic and execution information [56]. Business workflows focus on control flows, and data are processed by machines. Scientific workflows are more abstract focusing on data flows based on data dependencies and processed by humans or machines [19]. Workflow is denoted by DAG where tasks are represented by vertices and edges depict dependencies. Through efficient workflow scheduling, optimal utilization of resources can be achieved.

Workflow scheduling can be separated into—(1) scheduling process, (2) scheduling tasks, and (3) scheduling resources [51]. Process scheduling focuses on scheduling criteria and techniques for schedule generation. Task scheduling constitutes mapping of tasks to target machines. Liu et al. presented Voltage Island Largest Capacity First (VILCF) algorithm to schedule periodic tasks on a multi-core processor. The algorithm was based on DVFS and outperformed the existing algorithms for the multi-core voltage island [57]. Precedence constrained scheduling of parallel tasks on many-core processors was carried out by Keqin Li. Comparison with optimal algorithms proved its effectiveness [58]. Global task scheduling approaches used for prediction were discussed for mapping of tasks to their desired targets [21]. In resource scheduling, the execution model (public, private, and hybrid) and provisioning model are considered. Thread scheduling is necessary for the tiled multi-core environment to compensate for thread sensitivity toward shared resources [59]. Resource management was done through scheduling in the MapReduce model of cloud service (Cura) for providing cost-efficient services [60].

As per the literature, significant work is done on resource provisioning and task scheduling because these are the main steps to execute a workflow in CC [13].

4.1 W4 approach

W4 (What, Where, When, Why) approach is used to define all aspects related to energy-aware scheduling.

4.1.1 What is scheduling?: an NP-hard/complete problem

A problem is a decision problem if its solution is either “yes” or “no.” Scheduling tasks on target systems is a decision problem [57]. Decision problems are of following types:

P class—A problem is said to be in P class if it is solvable in polynomial time means O (n^m) in the worst case, where m is a constant. NP class—A problem is said to be NP if for instances where answers are yes can be tested in polynomial time. In this category of the problem, it is easy to check the correctness of the solution. NP-Hard class—There are some problems that cannot be solved directly; they are reduced to other problems. In addition, if the time taken for reduction is polynomial times, then they are reducible. If in NP class, all problems can be reduced in polynomial time to the other problem then they are NP-Hard. Also, for an NP-Hard problem, it is not compulsory to be in NP. In other words, if an algorithm takes huge time and is not feasible, then it is NP-Hard. NP-Complete class—Those problems which are NP-Hard and reside in NP Class too are NP-Complete [61].

There are different perspectives of researchers regarding scheduling. It is NP-Hard as mentioned in [12, 62]. However, scheduling is classified as an NP-Complete problem in [63, 64]. Therefore, this creates confusion. According to our view, the problem is NP-Hard if one can design a solution algorithm, but its execution takes huge time and is not feasible, e.g., if one maps a very large number of tasks say 5,000,000 tasks on computer systems in a certain short deadline, an algorithm can be designed, but it would not be feasible to schedule, i.e., NP-Hard. If one has to schedule 10 tasks, then this is possible, so it becomes NP-Complete.

4.1.2 Where energy-aware scheduling?

Energy-aware scheduling can be deployed with multi-core architectures, virtualization techniques, bio-inspired techniques, power-aware techniques, thermal-aware techniques. In all of these methods, scheduling algorithms are used as discussed in Sect. 2.2. Therefore, energy-aware scheduling becomes a holistic solution for achieving energy efficiency in CC.

4.1.3 When energy-aware scheduling?

Scheduling is based on criteria, which focus on one or more parameters [13]. These parameters can be objective or subjective. Objective parameters are measured directly, and output is numerically specified, e.g., time, cost, energy consumption, etc. Subjective parameters cannot be measured directly, e.g., fault tolerance can be measured through reliability. Energy-aware scheduling is done based on energy or power consumption criteria. Various parameters used for achieving energy efficiency are discussed in Sect. 5. Scheduling is classified as follows:

4.1.3.1 Best-effort scheduling (MIN–MAX)

In this scheduling, only one objective is focused upon without considering other objectives such as QoS factors, e.g., considering time as the only constraint without focusing on cost/energy or only energy-aware scheduling without focusing on QoS (minimizing SLA violations). The current state of the art in achieving energy efficiency in CC has limited focus on QoS [24, 65, 66].

4.1.3.2 QoS-constrained workflow scheduling

This type of scheduling is more used in actual applications. There is always a trade-off; if one tries to minimize certain factor, other automatically increases. Therefore, to handle the trade-off, it is done. The aim is to optimize one parameter while applying the constraint to another parameter. The goal is to generate a schedule in accordance with the preferable parameter meeting specified QoS constraints. Verma et al. presented a heuristic for scheduling workflow tasks having the budget and deadline constraints. The valuable trade-off was found between execution time and cost under these constraints. The simulation was performed with synthetic workflow applications to test the efficiency of presented heuristic. Results confirmed that offered heuristic decreased the cost keeping makespan as low as possible [67]. In another work on spectrum sensing, energy consumption was minimized by developing energy-efficient methods for body sensor networks while keeping satisfactory sensing quality [68]. In deadline-constrained workflow scheduling, one tries to minimize the cost while fulfilling timing constraints. Netjinda et al. focused on optimization of the cost of IaaS cloud services while executing scientific workflow within particular deadline constraints. Swarm optimization techniques were used. The results showed improvements in comparison with other algorithms by decreasing the total cost [34]. In budget-constrained workflow scheduling, execution of the workflow is completed while maintaining budget constraints. Kumar et al. proposed a scheme for SLA negotiation for budget, energy, and time. Authors made a strategy for making a cost-effective schedule without sacrificing performance. The simulation was performed for the evaluation of the proposed scheme which indicated it is worth [35].

4.1.3.3 Multi-criteria workflow scheduling

In this type of scheduling, many parameters are considered simultaneously which conflict often. It could be QoS-constrained even. In [16], VM consolidation was performed using prediction algorithms. Novel multi-criteria techniques were employed for selection of overloaded hosts and appropriate VMs. Results showed 98.11% reduction in a metric composed of migrations, violations in SLA, and consumption of energy.

Aggregation approach uses a simple average of an objective function to select a final solution. Mukhopadhyay et al. [69] reported a work that performed the final selection of solution based on aggregation function and optimization of aggregated fitness function.

£-approach User does not always know to keep a certain constraint for one criterion. Therefore, mostly for solving a bi-criteria problem, this approach is used [70].

4.1.3.4 Pareto approach

In this approach, solutions cannot be more optimized across any dimension without being deteriorated across another dimension at the same time [71]. Solution set obtained after using the Pareto approach is called Pareto optimal solution. In [72], two scheduling problems based on Pareto optimization were investigated. The maximum earliness cost was the objective in the first and in the second, maximum earliness cost was objective for one agent, and total earliness cost was objective for a second agent. As per authors, problems could be fixed in polynomial time by predicting Pareto optimal points.

4.1.4 Why energy-aware scheduling?

Energy consumed by IT equipment (in data centers) is increasing at an extremely high rate, and release of GHGs from them is making earth unfit to live. Energy-aware scheduling is ubiquitous and must be used for a sustainable future. Otherwise, it will have adverse effects on the environment. For optimal use of resources (in data centers) and to achieve QoS, energy-efficient scheduling plays a major role. Underutilized resources being idle consume power at leisure, and on the other side, overutilized resources degrade the performance.

5 Optimization: a solution

Scheduling is a decision problem as discussed in Sect. 4.1.1, so it cannot have a precise solution. Heuristics-based optimization techniques can assist in generating an optimal solution, where the objective is either to minimize or maximize a certain parameter. Parameters are finalized as per criteria which could be single or multi-objective.

To solve the energy-aware scheduling problem (optimization problem), various methods/techniques are available. Energy-aware scheduling focuses upon energy and/or power as a criterion for scheduling. According to Beloglazov and Buyya [11], the rate at which a system carries out work is power, whereas energy is total work done at a definite time interval as illustrated in Eq. (1).

$${\text{Energy}} = {\text{Work}} = {\text{Power}}*{\text{Time}}$$

(1)

According to [11], a decrease in the consumption of power does not always decrease energy consumption. For example, if there is a decrease in power consumed (decreasing CPU’s load), then the program may take longer time thus consuming more energy. Power is of two kinds—(1) static and (2) dynamic. Static power or leakage power is power consumed by the system not in functioning state. It depends upon low-level system design, i.e., transistors and processor technology. Therefore, it is difficult to reduce. Dynamic power depends upon usage scenarios, the voltage that is supplied and frequency of the clock. It can be reduced by reducing voltage and clock frequency. Power is a significant design constraint for computing systems. Power efficient computing is the main focus during research. However, due to the close correlation, energy and power are used interchangeably in the literature. The survey classifies various methods for reducing power or energy consumption.

5.1 Dynamic power management methods for optimization

The methods for dynamic power management are shown in Fig. 7.

Fig. 7

Methods for dynamic power management

Dynamic performance scaling DVFS reduces power consumption by frequency scaling (up/down) according to the requirement. If a CPU runs at a lesser frequency, less voltage will be required and hence less power consumption. Therefore, voltage and frequency are balanced dynamically, resulting in decreased power consumption. It may take longer time if a processor is working at lower frequencies. Processor’s governor monitors this process to regulate the performance. Generally, the processor starts at lower frequencies and steadily increases with workloads and so is energy consumption. A major challenge is to reduce power consumption through DVFS while handling deadline constraints. In [73], authors tried to decrease energy consumption by making CPU work at lower frequencies as long as task deadlines could be guaranteed.

Slack reclamation is a technique that can be used with DVFS in order to meet timing constraints for completion of tasks. In parallel processing, many tasks execute simultaneously. Further, if completion of a task depends upon two preceding tasks and these two tasks complete at dissimilar times, a task that completes earlier can manage addition runtime called as slack. This additional time can be used by DVFS for energy efficiency [74].

Dynamic Component Deactivation (DCD) It involves deactivating or shutting down components, which are at idle (not used) state. DCD has transition overhead, which is insignificant in the case of small problems. However, such transitions can cause performance degradation and delays drawing additional power in some cases. Therefore, a transition is only required when the idle timing period is sufficient to pay off for overhead during transitions. In real life scenarios, it is impossible to predict future workload. Therefore, an estimation of actual transition requires historical data or some system model [75, 76].

Table 5 compares power reduction techniques. Most of the work is based on benchmarks or synthetic datasets in contrast to real data. Most commonly used simulators are CloudSim, Sniper, McPat in contrast to testbed or the experimental environment. DVFS in combination with other techniques is used for reducing power consumption, and most results are objective in nature (numerically specified). Future work includes implementation of the real-time environment while scheduling more number of jobs, considering platform heterogeneity, network characteristics, communication costs, etc. The criteria or objective is to decrease power based on some parameters (objective or subjective). Various parameters used in power scaling are:

Table 5 Comparison of power-aware scheduling techniques

Sudhanshu et al. [63] developed a hierarchical framework for the management of power on many-core tiled processors. Weighted speed up, system throughput, cache-to-cache transfers were employed to improve system performance with DVFS and a special thread scheduler. In [77], parameters targeted were servers used, energy consumption, cumulative machine uptime (CMU) for energy savings while guaranteeing SLAs. A heuristic based on DVFS was proposed to combine different virtualized clusters on physical machines for batch-oriented cases. Morteza and Mehdi [78] provided an approach for analyzing and modeling of the real system serving stochastic workloads. Accuracy in terms of the average temperature of each core, thermal parameters including ambient temperature, convection resistance, convection capacitance, and others including relative error, and absolute error were focused upon to reduce power consumption. In [79], power consumption, performance overhead, execution time, CPU utilization were the parameters used. A scheme was devised which could estimate and regulate power consumption and its impact on performance to suit power capping schemes.

In [80], parameters were energy performance ratio, power, runtime performance, energy consumption, and energy delay product (EDP) index. The major thought was to examine latency characteristics. In [55], energy consumption per cycle, time and energy overhead parameters were focused upon. Energy-aware dynamic task scheduling (EDTS) algorithm was developed to test online communications between tasks and reduce the overall consumption of energy. The designed algorithm made use of static scheduling algorithm’s results and blindly minimized consumption of energy. In [73], cloud servers’ performance, VM overhead, resource, and CPU utilization were the parameters employed for improving performance in a cloud data center and also to make it energy efficient. Babukarthik et al. [81] focused on the number of processors, tasks, and speed of execution to minimize energy consumption while scheduling tasks. A hybrid algorithm was proposed with the advantage of ACO, cuckoo search, and voltage scaling. In [20], deadline, VM overhead, performance metrics, makespan, energy consumption, energy cost, CO₂ emission, provider’s performance, the resource utilization rate were examined maintaining SLA and QoS constraints. The problem of scheduling scientific workflow applications in a time-dependent environment was addressed.

In [82], makespan, resource utilization, given deadline were the parameters to minimize the total energy consumption of scientific workflow. Tasks were allocated to heterogeneous machines having a deadline and different frequency capabilities. The proposed algorithm worked repeatedly for further scaling of frequency. In [83], performance, energy-related metrics were taken into consideration to reduce energy consumption. Authors presented a code scheduling approach that used DVS and power gating. The objective was to minimize consumption of power during application execution.

In [84], makespan, energy, schedule length ratio (SLR), and energy consumption ratio (ECR) were focused upon to address the problem of scheduling of precedence constrained parallel applications on multiprocessor computer systems. Algorithms with the incorporation of relative superiority metric (RS) and makespan-conservative energy reduction technique (MCER) greatly contributed to reducing energy consumption. The energy saving of energy-conscious scheduling (ECS) and EC_{+ idle} was enabled by making use of the DVS technique. In [85], deadline constraints, execution time, and speed were the parameters used. An online scheme was presented to allocate speeds to hard real-time workloads on systems facing thermal problems. In [86], system performance, core utilization of active nodes, and node utilization were focused upon to target interference that applications experience at an inter-core granularity. Authors presented a model for improving system performance using slack-based scheduling. In [87], DVFS improved makespan by more than 65% and at the same time improved the dynamic energy by about 20%.

From the above-mentioned literature, it is found that the number of processors, servers, cores, resources, nodes, and CPU utilization are the most preferred parameters used in power scaling methods. It is considered to be the most favored technique for hardware-based optimization as the frequency of publications in this domain has increased in the last years (Sect. 7).

5.2 Heuristics-based optimization methods

Software-based optimizations include resource scheduling, allocation techniques, throttling, and use of parallel programs. Being an NP-Hard problem, scheduling is performed by using various software-based optimization techniques—heuristics and metaheuristics. Figure 8 shows the classification of optimization techniques used in energy-aware scheduling.

Fig. 8

Classification of heuristics-based optimization methods/techniques

Heuristics are usually dependent on the problem type, whereas metaheuristics do not depend on the type of problem and can be applied to any problem [88]. Heuristics are generally based on the local search.

Metaheuristics are the advanced version of heuristics based on the generalized local search or global search and can be thought of guiding principle to design underlying heuristics [89].

Approximate Algorithms Heuristics have less time complexity than traditional methods. The aim of heuristic is to yield rapidly a solution that is decent enough for resolving the problem. Heuristics are based on theories or experimental experiences, but approximation algorithms have a solid theoretical foundation.

Global search [90, 91] Heuristics are generally based on local search, whereas the metaheuristics do not get stuck in local optima as they use generalized local search or global search. In a global search, whole solution space is searched to find a minimum or maximum value. Generally, methods used in global search do not use greedy approach because entire search space needs to be examined and globally best is selected. They could utilize dynamic methods.

Local search Local search finds a minimum or a maximum solution in local space. Local search algorithms find an estimated optimal solution out of possible candidate solutions by moving from one solution to others within a time interval. These algorithms may use greedy approaches. Works on the local search are [90, 92, 93].

Greedy methods Greedy methods make the best choice at a particular instant of time. Thus, the local optimal solution is selected with an expectation that it would lead to a global best solution. At each step, greedy decisions are made to ensure the optimization of an objective function. The greedy algorithm cannot go backward to change the decision. [94,95,96,97] use greedy algorithms for scheduling.

Dynamic methods In dynamic programming, the problem is distributed into smaller problems, and each smaller problem is solved only once, resulting in a decrease in the number of computations. The solution is stored for next time usage. This method is advantageous when there is an exponential growth of repeating subproblems as a function of the size of the input. Authors in [98] used dynamic programming to find an approximate method for energy-efficient scheduling. Both dynamic and greedy approaches can be applied to the same problem; the difference is that the greedy approach does not reconsider its decision, whereas dynamic approach may keep on refining choices. Heuristics can be implemented independently or in combination with other optimization algorithms to give better efficiency.

Methods based on these techniques are described as follows:

A deterministic model produces a single outcome at every instant based on all given input values. Deterministic methods employ state space search algorithms. Probabilistic methods are based on randomness for the accomplishment of objectives.

State Space Search visits the entire space to reach a solution following certain rules. Informed search use heuristics (a function whose result indicates the next move). A branch and bound algorithm selects the optimum answer of an optimization problem. The entire space of the solution domain is examined for searching the best solution. The limits in objective function are merged with the last best solution. It continues to improve the solution once originated. Parts of solution space are found completely keeping the path with the lowest cost as a target [98, 99]. In the state space search, many states are traversed to reach a final state or goal state. Sequences of actions, which lead to the goal state, constitute the solution [100]. In hill climbing, the search continues in the direction which optimizes the cost using a greedy approach. There are many variants of hill climbing—the best neighbor, the first or nearest neighbor. The best-first search uses an evaluation function based on heuristic to explore the graph. The best-first search can be optimized to reduce the memory required and is called the beam search.

Table 6 compares the state space search (informed search) deterministic optimization methods used in energy-aware scheduling. Maximum datasets are either randomly generated. Mostly simulations are performed, and the results obtained are numerically specified (objective). Future scope includes increasing the complexity of networks and consideration of system components (disk, main memory, and communication networks) for energy efficiency. The objective is to decrease energy consumption based on some parameters (objective or subjective) that are as follows:

Table 6 Comparison of state space search (deterministic) technique

In [101] nominal execution time, nominal system utilization, system slackness, resource utilization were the parameters used. The focus was on the allocation of resources and robustness based on QoS constraints. Szynkiewicz et al. focused on traffic difference, power reduction, and QoS constraints. Design of a framework for centralized and hierarchal variants for a low energy-consuming network was made. Two control levels were implemented, network-based mechanism and local mechanism. Network-wide optimization problem was formulated in two ways, and an efficient algorithm was developed to solve it. Total system runtime, utility, QoS, average energy, and power consumption were the parameters used in [43]. A general adaptive task model was presented by utilizing existing ways of real-time adaptation for fault tolerance and graceful degradation [102]. Optimal solutions using heuristics were presented to get maximum advantages within the limited energy budget and a known time to recharge. A design space exploration (DSE) [103] method was developed to present architecture having multiple cores and optimal scheduling. Its efficiency was proved with large and hard graph problem. In [104], the technique used was Spreading Activation Partial Order Planner (SA-POP). Other techniques based on precedence constraints were applied to find harms and enhance ordering restrictions for autonomous coordination.

It is found that nominal execution time, total system runtime, utility, number of cores, number of buses, system utilization are the most preferred parameters in deterministic methods. As per the literature, deterministic algorithms are generally used with power scaling and genetic techniques.

Queuing Theory: It comes under the category of probabilistic methods. In the queuing theory [105], queues are presented and analyzed. Construction of models is done to estimate the length of queues and their waiting time. Queues are represented in mathematical equations for proving theorems known as Markov chains. Various scheduling policies can be used at queuing nodes and represented mathematically. First come first-serve (FCFS) [89] algorithm schedules processes by managing tasks or resources in order of their arrival times. It works on the principle of first in, first out (FIFO). The other algorithm is last in, first out (LIFO). It serves the task which has shorter waiting time first. Minimum completion time (MCT) and minimum execution time (MET) [106] are two heuristic algorithms. Min–Min and Max–Min are also two heuristics based on MCT and MET. Min–Min [106] picks the machine which gives MCT and assigns the smallest task to that machine. It increases the value of makespan, but does not consider the availability of resources while scheduling. Thus, completion and execution time come to be almost the same [106]. Other algorithms include priority-based scheduling using priority queues where jobs are executed on the basis of size, time, etc. These include EDF, Shortest Job First (SJF), Earliest Finish Time (EFT), Heterogeneous Earliest Finish Time (HEFT), etc.

Table 7 compares queuing theory techniques based on dataset, tools, techniques, results, and future scope. Mostly synthetic datasets and real-world traces are used for evaluation. Techniques such as FCFS, HEFT, EDF are used, and most of the results are objective.

Table 7 Comparison of techniques using queuing theory in energy-aware scheduling

In [107], energy consumption, slack factor, good state probability, bad state probability, number of nodes were the parameters used for scheduling of periodic messages and to decrease overall consumption of energy in a wireless network. Thanavanich and Uthayopas [108] used a metric called SLR and ECR to balance energy consumption and makespan simultaneously. Two energy-efficient cloud-based scheduling approaches called Enhancing Heterogeneous Earliest Finish Time (EHEFT) and Enhancing Critical Path on a Processor (ECPOP) were proposed. They tried to achieve more energy reduction and satisfy performance constraints. The proposed approach used performance metric ratio of effectiveness (RE) to find a processor which is ineffective. In [82], makespan, utilization, user deadline were the parameters used to allocate tasks on heterogeneous machines with a deadline and diverse frequency capabilities. Authors relied upon the fact that even the minimum frequency may not always prove to be energy efficient.

In [74], makespan, energy, SLR, and ECR were the parameters used on multiprocessor computer systems. Algorithms with RS and MCER significantly contributed to reducing energy consumption. In essence, an energy saving of ECS and EC_+idle was enabled due to the exploitation of the DVS technique. In [85], deadline constraints, execution time, and speed were parameters used. An online scheme was presented which considered deadline constraints while providing speeds to systems processing real-time workloads having thermal issues. In [34], execution time, makespan, and total cost were used to optimize the cost of purchasing IaaS to achieve scientific workflow execution within specific deadlines. Authors used PSO along with VNS to optimize numerous factors such as a number of machines, price, scheduling to minimize total cost. Jingcao and Marculescu et al. [109] developed an effective scheme for energy-aware scheduling, which considered a delay in communication by parallel scheduling of transactions with computation for Network-on-Chip (NoC) architectures. They also handled execution time and cost. Zhang et al. [110] simulated the thermal-aware task scheduling algorithm and thermal-aware task scheduling algorithm-backfilled based on thermal information and resource information obtained in the Center for Computational Research (CCR) log files. Data center average, maximum temperature, job response time, impact on the environment, consumed power were measured.

It is clear that QoS parameters, benchmark programs, and real data are also focused upon while reducing energy consumption. EFT and other algorithms based on EFT are most widely used algorithms in queuing theory. Network characteristics, power consumption by hardware components, and cost of data transfer are not considered significant in queuing theory methods.

Bin packing is a probabilistic heuristic technique which aims to switch off idle servers by packing the hosts on available VMs [111]. It involves VM migrations on physical machines to fulfill requests by utilizing a minimum number of servers [112]. It uses next-fit, first-fit and best-fit scheduling algorithms. The next fit algorithm utilizes the same bin, used during the last processed item. First fit is a modification of the next fit; it examines earlier bins and selects the best. Best fit selects bin with minimum space wastage and is considered to be finest. First fit decreasing (FFD) and best fit decreasing (BFD) are modifications of these algorithms that are based on sorting.

Table 8 compares bin packing-based energy-aware scheduling techniques. The data include real measurements, real-time traces, and synthetically generated datasets. Experiment setups include either hardware devices or simulators. Techniques include MBFD, minimum migrations (MM), best-fit heuristics.

Table 8 Comparison of bin packing-based energy-aware scheduling techniques

Taheri et al. [113] tried to decrease the consumption of energy in CC data centers revising VM scheduling method while keeping QoS parameters as high as possible. In [114], an investigation was performed to find performance assurances that could be severely proven for heuristics (MM and MBFD) according to capacity and number of hosts to indicate the effectiveness of approximation. Viswanathan et al. [115] proposed a novel resource heuristic framework that used a best-fit heuristic for reallocation of unfinished tasks to alternate (backup) service providers. Zeng et al. [116] targeted energy-efficient scheduling of real-time periodic tasks. Some constraints like idle power, ineffective speed, and application-specific power characteristics, etc., were also associated with them. An adaptive minimal bound first-fit (AMBFF) algorithm was proposed for both dynamic-priority and fixed-priority multiprocessor.

In [117], the objective was to provide a computational cost and performance benefit analysis of schemes in terms of both feasibility and overall energy consumption. Experimental evaluation was performed to check the impact of partitioning heuristics, admission control algorithms, and speed assignment schemes by introducing a hybrid metric. Kandhalu et al. [118] studied energy-efficient scheduling of periodic real-time tasks with implicit deadlines on-chip multi-core processors using normalized power consumption to indicate its performance. In [77], parameters targeted were servers used, energy consumption, cumulative machine uptime (CMU) for energy savings while guaranteeing SLAs. In [73], cloud servers’ performance, VM overhead, resource, and CPU utilization were used to improve performance and energy efficiency. In [119], Hancong et al. proposed a scheduling approach named pre-ant policy. It consisted of a model for prediction using mathematics and a scheduler using improved ant colony algorithm.

In bin packing techniques, power is not aggressively reduced without considering performance. Most used algorithms are best-fit heuristics and modified best, and worst fit algorithms.

Metaheuristics are heuristics about heuristics, and they provide better results by avoiding local optima. They can be based on local search, global search, and both. So, there is an overlapping classification as shown in Fig. 8. There are many classifications related to these heuristics/metaheuristics-based optimization techniques. Figure 8 lists only algorithms related to energy-aware scheduling in CC. They can be put under multiple classes like swarm intelligence in nature-inspired, or Tabu search can come under memory-based metaheuristics. Therefore, to explain all the possible permutations is beyond the scope. So, algorithms used in energy-aware scheduling are covered and classified in the best possible way.

Trajectory-based methods use a single search strategy, and they focus on a single candidate solution. Trajectory-based algorithms that use both local search and global search are simulated annealing, Tabu search, iterated local search, variable neighborhood search, greedy randomized adaptive search procedure (GRASP). However, algorithms that use global search are generally population-based metaheuristics which use multiple candidates as search points, and characteristics of the population are used to guide the search. Examples are ant colony optimization (ACO), particle swarm optimization (PSO), and cuckoo Search.

Iterated local search is an improvement of local search in which search is repeated each time starting from a different state using certain criteria until the best solution is found. It helps to avoid getting stuck in local optima. Memory might be used to keep a record of previously visited states. Variable neighborhood search is based on dynamically changing neighbors. Random neighbors list can be made, but the certain sequence is to be followed. Iterated local search is applied and move to the next state is possible only if a better solution can be generated. Simulated annealing is inspired by metallurgical annealing. It is the type of trajectory-based metaheuristic that is used to generate global optimum in the large search space as per the objective function.

Tabu search uses improved local search and memory to avoid cycles. Memory records recently visited states and prevented moving toward them. The best state that has not been visited yet is chosen in each iteration until the algorithm is stopped at a terminating condition.

GRASP generates solution using dynamic constructive heuristic and randomization. The next state is chosen at random. It continues to improve the solution until the best is found.

Table 9 compares (metaheuristics) techniques based on trajectory methods. In [120], completion time, disc utilization, processor utilization, the power consumed were used for scheduling of VMs to physical machine considering energy efficiency, synchronizing utilization of the processor, disc, and cost of migration. In [121], the number of migrations, performance degradation due to migrations, SLA violations were the parameters considered to evaluate the proposed framework that consolidates VMs while taking care of QoS. Alkhashai et al. [122] considered makespan, utilization of resources and, cost for scheduling of tasks in a cloud environment. The proposed algorithm was able to reduce time, cost, and increase resource utilization. In [123], to maintain energy performance trade-off, a mechanism was introduced that focused on probability functions and a number of cycles generated to reduce carbon footprints.

Table 9 Comparison of trajectory-based energy-aware scheduling techniques

It can be seen that CloudSim and MATLAB are the most widely used tools in the case of trajectory-based methods and results are objective in nature. However, real cloud implementation is lacking behind.

Population-based metaheuristics are further divided into evolutionary algorithms and algorithms based on swarm intelligence.

Evolutionary algorithm (EA) EA is a subgroup of evolutionary computation, which is based on metaheuristic technique. It uses various nature-inspired mechanisms such as reproduction, mutation, and crossover. Genetic algorithms [88, 124] are a most popular type of EA.

Table 10 compares techniques based on genetic algorithms. In [125], for efficient scheduling, Multi-objective genetic algorithm (MOGA) was proposed which considered both optimization and global makespan to reduce energy consumption. Kołodziej et al. [126] formulated task scheduling for CC as a bi-objective minimization problem having makespan and energy consumption as criteria and QoS constraints. Youness et al. [103] focused on average deviation from optimality, number of cores, execution time, and scalability. A design space exploration (DSE) methodology was proposed to create architectures with multiple cores and optimal scheduling. In [127], cost of storage, computation, and data transfer were considered. In [128], performance, resource utilization, number of servers were considered. To reduce the number of running servers and resource wastage, a hybrid algorithm was proposed. In [129], parameters that were focused upon included makespan, VM resource utilization, the degree of imbalance, performance.

Table 10 Comparison of genetic algorithm-based energy-aware scheduling techniques

Few works on evolutionary algorithms have actually focused on QoS parameters. Researchers employed various techniques based on genetic algorithms for saving energy, but results cannot be compared or correlated because of different constraints and scenarios.

Swarm intelligence [130] is based on mutual behavior of a population of agents occurring in nature. It motivates from the behavior of animals as a group, how they interact and communicate among themselves. Metaheuristics-based swarm intelligence algorithms for scheduling include (1) ACO [131] which is inspired by the behavior of ants to discover the shortest path to the source of food. (2) PSO [131] is motivated by the social behavior of particles. PSO generally combines local search and global search methods for resource allocation. Cuckoo search [132] is motivated by blood parasitism of cuckoo species. Table 11 compares swarm intelligence-based energy-aware scheduling techniques on the basis of dataset, tools, techniques, results. Synthetically generated dataset and real-world traces are used for evaluation. Techniques like PSO, ACO, cuckoo Search are employed, and most results are objective. The main aim is to save energy by using different parameters.

Table 11 Comparison of swarm intelligence-based energy-aware scheduling techniques

Babukarthik et al. [81] focused on quality of schedule, number of tasks, number of processors, and speed of execution to minimize energy consumption while scheduling tasks. In [133], makespan, cost, job rejection ratio, number of jobs meeting the deadline, and user satisfaction were considered. Authors designed and developed CLOUD Resource Broker (CLOUDRB) for effective management of resources in the cloud. Jeyarani et al. [134] proposed an adaptive power-aware virtual machine provisioner (APA-VMP) which drew minimum power without compromising performance. In [135], a strategy for scheduling and resource provisioning for different workflows on IaaS was presented. It optimized application execution cost maintaining deadline constraints. In [34], execution time, makespan, and the total cost were the parameters used to optimize the cost of purchasing IaaS to achieve scientific workflow execution within specific deadlines. In [119], Hancong et al. proposed a scheduling approach named pre-ant policy. Faragardi et al. [22] presented a scheme for the allocation of resources considering energy efficiency, reliability, timing constraints, memory limitation, etc. Reliability and quality of schedule were also considered.

Most used algorithms are PSO and ACO which helped to satisfy the goal of reducing energy consumption.

Miscellaneous techniques Table 12 compares these techniques. Experimental setup includes real cloud deployments; CloudSim is the most used simulator. Most of the result analysis is objectively discussed. Future work includes considering more constraints, implementation in a real environment, and developing a generic framework for energy savings.

Table 12 Miscellaneous techniques

6 Performance metrics for green cloud computing

Performance metrics for the green data center are categorized as basic metrics and extended metrics [46]. The basic metrics are a measure of environmental friendliness of data centers. The extended metrics are functions of basic metrics that generate detailed view about data center [49].

6.1 Basic metrics

These are used to illustrate the efficiency of data centers in terms of environmental effect.

6.1.1 Greenhouse gas emission

GHGs are gases in the atmosphere that absorb and produce heat rays in the thermal infrared range. GHG is CO₂ which constitutes 9–26% of effect [141]. Power consumed in data centers is enormous, and GHGs are released during power generation which causes harsh effects on the environment. Thus, GHG emission should be measured to check how green a data center is.

6.1.2 Humidity

Moisture content in the air is called humidity [142]. Hardware failures are caused due to high humidity. The quantity of water in the air is measured by relative humidity. Relative humidity difference (RHD) (Eq. (2)) is the difference between the relative humidity of return air and air supply in the data center:

$${\text{RHD}} = {\text{Return}}\,{\text{air}}\,{\text{relative}}\,{\text{humidity}} - {\text{Supply}}\,{\text{air}}\,{\text{relative }}\,{\text{humidity}}$$

(2)

6.1.3 Thermal metrics

They play a significant role in maintaining the data center’s efficiency.

6.1.3.1 Data center temperature

To attain system reliability, an optimal temperature range is between 20 to 24 °C [143]. If data center temperature is higher than 30 °C [144], it is recommended that no costly IT equipment should be kept there.

British thermal unit (BTU): Solutions for cooling a server room or a data center are governed by British Thermal Unit (BTU) [145]. A BTU is the measure of energy that is needed for increasing temperature of a pound of water by 1 °F.

Airflow performance index: It indicates the efficiency of the data center from a thermal point of view.

Cooling system efficiency metric: These include airflow efficiency (AE), cooling system efficiency (CSE), cooling system sizing (CSS), and water economizer utilization (WEU). AE (Eq. (3)) indicates how efficiently air passes through a data center.

$${\text{AE}} = 1000 \times \frac{{{\text{The}}\,{\text{overall}}\, {\text{power}}\, {\text{of}}\, {\text{fan }}}}{{{\text{Overall}}\, {\text{airflow}}\, {\text{of }}\,{\text{the}}\,{\text{fan }}}}$$

(3)

CSE (Eq. (4)) is a measure of total efficiency in terms of cooling equipment usage, power withdrawn for cooling. It is defined as:

$${\text{CSE }} = \frac{\text{Average power used by cooling systems }}{\text{Average load to be cooled}}$$

(4)

CSS (Eq. (5)) is a ratio of installed chilling capacity to highest chilling load [146].

$${\text{CSS}} = \frac{\text{Installed capacity for cooling}}{\text{Highest load for cooling}}$$

(5)

WEU (Eq. (6)) is a measure of savings in energy by using a water-sider economizer system.

$${\text{WEU}} = \frac{\text{Water economizer hours}}{{\begin{array}{*{20}c} {24 \times 365} \\ \end{array} }}$$

(6)

WEU provides information on energy savings by using a water-side economizer system.

6.1.4 Power/energy metrics

They include Data Center Infrastructure Efficiency (DCiE or DCE), Power Usage Effectiveness (PUE), Heating, Ventilation and Air Conditioning (HVAC) System Effectiveness, Space, Watts and Performance (SWaP), Data Center Energy Productivity (DCeP). DCiE is widely accepted by industry [147, 148]. DCiE is calculated as mentioned in Eq. (7).

$${\text{DCiE }} = \frac{\text{Power consumed by IT devices}}{\text{Overall facility power}}$$

(7)

PUE [147] measures the energy consumed by IT and non-IT equipment (cooling devices). It is defined in Eq. (8).

$${\text{PUE}} = \frac{1}{\text{DCiE}} = \frac{{ {\text{Overall facility power}}}}{\text{Power consumed by IT devices}}$$

(8)

The HVAC System Effectiveness (Eq. (9)) is a ratio of energy consumed by IT devices to HVAC (electrical energy for cooling, movement of a fan) system energy.

$${\text{HVAC Effectiveness }} = \frac{\text{Energy consumption by IT devices}}{{{\text{HVAC}} + \left( {{\text{Steam}} + {\text{Chilled Water}} + {\text{Fuel}} } \right) \times 293}},$$

(9)

where IT is total electrical energy consumed annually by IT devices. HVAC, fuel, steam, chilled water is annual electrical energy required for HVAC, fuel, steam, chilled water, respectively.

SWaP (Eq. (10)) measures energy efficiency by considering space, energy, and performance together [149].

$${\text{SWaP }} = \frac{{ {\text{System Performance}}}}{{{\text{Rack Space}} \times {\text{Consumption of Power}}}}$$

(10)

where performance is measured by using benchmarks set by the industry. Space is a measure of the height of the server in rack units. Power is in watts, which is used by the system during benchmark runs.

The DCeP (Eq. (11)) [150, 151] measures useful work done in comparison with consumption of energy.

$${\text{DCeP }} = \frac{\text{Useful work done}}{\text{Total energy consumption}}$$

(11)

6.2 Extended metrics

Extended metrics give detailed information about the data center and are categorized into multiple indicators and total cost of ownership.

6.2.1 Multiple indicators

Multiple indicators include data center indicators and data center sub-level indicators.

Data center indicators include server utilization, network utilization, storage utilization, and data center utilization.

Server usage measures actions of the processor in contrast to its maximum capability during uppermost frequency state. Network usage is the ratio of bandwidth consumed to bandwidth capacity in the data center. Storage usage is a percentage of storage consumed compared to total storage within the data center. Data center utilization indicates the amount of power consumed by IT equipment comparative to the real capability of the data center.

Data center sub-level indicators Power, cooling, airflow, weight, and area constitute sub-level indicators [147]. They help to measure various inefficiencies in data centers.

6.2.2 Total cost of ownership (TCO)

TCO signifies cost required to buy, construct, run, and maintain a data center [152]. TCO includes capital expenses which include initial investments, cost of cooling equipment, power consumption, space, and operational expenses, etc. So, the cost of the data center should be in terms of dollars per watt. Generally, power and cooling forms 80% of the capital cost and rest 20% is spent on construction of data center [153]. Operational cost includes monthly costs of running a data center [153], e.g., implementation techniques, climate costs, etc.

The studies that have measured the values of these metrics to find the efficiency of data centers are [37, 146].

7 Global analysis

The rapid development of CC has escalated the need of energy efficiency. According to the best of our knowledge, there is a scarcity of broad scientometric analysis (empirical study) which provides a view of the present status of research in this domain. There is a need to explore the trends of research in this evolving field. Scientometric analysis [154] has been conducted to find global research trends which can serve as a direction for further research activities, collaborative research, sharing of knowledge. Publications mentioned in Sect. 5 are included in the analysis to examine research in this field. This survey evaluates the publications according to global regions, journals, conferences, year of publishing, the research community, and fund provisioning. The discussion below is based on trends found from this analysis. Peer researchers may interpret the trends differently.

Figure 9 illustrates the frequency of publications in various conferences and journals on energy-aware scheduling in CC. Journal considered are “Journal of Supercomputing (JSC), Journal of Parallel and Distributed Computing, Future Generation Computer Systems (FGCS), IEEE Transactions on Parallel and Distributed Computing (ITPD) and IEEE Communication Surveys and Tutorials (ICST).” Trends indicate that research work in this domain is published equally in journals and conferences. The reason may be the evolving and dynamic nature of the topic itself [10]. It can also be analyzed that FGCS and JSC are most preferred journals in the research community. Figure 10 illustrates a geographical viewpoint of this research domain. It is clear that Asia, North America, and Europe are more active research continents, whereas others contribute least. Asian people contribute a major part (43.33%) in this research domain.

Fig. 9

Journal- and conference-wise publications in all sub-domains in energy-aware scheduling

Fig. 10

Continent-wise publications in all sub-domains in energy-aware scheduling

Figure 11 illustrates a more detailed viewpoint by providing insights on publication details. European researchers prefer to publish a major part of their research contribution in conferences and FGCS. North America publishes a major part of research contribution in conferences. Asia prefers to publish in FGCS, JSC, and conferences, whereas Australia and Africa have the least contribution. African researcher prefers to publish in conferences. Australian researcher prefers to publish in ITPD and JSC. Figure 12 illustrates the frequency of publications annually in this domain. The year 2016 depicts maximum frequency (21.66%) of publications. Coming years may have an increase in contribution or publications. Table 13 shows the comparative analysis of published and funded work in all sub-domains. DVFS has a maximum number of publications which account for 23.33% of total publications. Out of this 23.33% of total publications, 13.33% have got grants. In miscellaneous field, 13.33% of publications are funded out of total 16.66%. Minimum funding is in a subdomain of trajectory-based methods in which only 1.6% works have got funding out of a total of 6.66%. Figure 13 illustrates the contribution in research by different groups in this domain. Academic research contribution is major in contrast to minor contribution from the collaborative effort of academia and industry. Research institutes of industry lack behind.

Fig. 11

Continent-wise publications according to journals and conferences

Fig. 12

Year-wise publications in all sub-domains in energy-aware scheduling

Table 13 Comparative analysis of published and funded

Fig. 13

Research contribution by different groups

8 Research issues and a strategy for future research

Energy efficiency in CC is an active and vast research area. Literature indicates that researchers are working on a particular sub-part of the solution. The problem of energy consumption is addressed by using various mechanisms such as virtualization, multi-core architectures, parallel processing, power-aware methods, thermal-aware methods, bio-inspired methods. These mechanisms in turn use various optimization/scheduling algorithms like PSO, ACO, HEFT, FCFS, best fit, genetic algorithm, etc. Limited research is carried on investigating the trade-off between energy efficiency and QoS compliance as per SLA. However, QoS is the foremost parameter of concern for cloud clients. To explore both performance and energy efficiency, the challenges include maintaining the reliability of a server because power cycling may reduce it, performing VMs consolidation without affecting QoS, accurate application performance management in a virtualized environment while maintaining SLA [65]. Major issues on energy optimization for CC are listed below:

• Much work is done on the development of energy-efficient framework using consolidation techniques, but that is not generic [77].

• How to balance energy efficiency and VM placement [66].

• Solution for VM performance degradation [121, 155].

• Implementation of optimization techniques for energy efficiency in the cloud environment taking into consideration a very high number of jobs [73, 81].

• Investigation of the impact of network characteristics, communication cost, overhead, system components like the disk, main memory while scheduling applications on cloud [82].

• The research in the area of harvesting renewable energy at different sites of a data center is at its initial stage. The challenge is to minimize nonrenewable energy usage, carbon cost and to investigate the effect of inter-region migration of VMs [156].

• Another issue is to select the VM that is to be migrated keeping in consideration the running application, SLA, data transfer, etc., and in some cases, multiple VMs are to be migrated. So, sharing of network resources effectively is a challenge [156].

• Performance management by synchronization with SLAs for the satisfaction of users [65].

• Minimizing energy consumption considering heterogeneous workloads and runtime migrations of VMs [156].

• Storing a large amount of data and its processing can lead to energy wastage. So, streamlining data storage, processing, energy consumption simultaneously is another issue [6].

A strategy for future research must address these issues. Work in harvesting renewable energy (data centers) is in its early stages. So a scheduling strategy employing heuristics-based optimization techniques that can migrate VMs to hosts of a green data center is needed. The solution should be generic and also take into account CPU utilization, RAM, network and storage devices. The new scheduling strategy should provide compliance to new paradigms of fog and edge computing by providing low-latency services to users so as to consider QoS factors (energy consumption and performance). Constraints associated with applications like speed, power in an idle state, power characteristics of an application [114] should be incorporated in the proposed strategy. The design strategy should also consider those applications in which execution time is not known beforehand [34].

9 Conclusion

CC has brought a revolution in today’s world by changing the way of delivering computing services. Almost all the online users use it in a direct or an indirect way. However, CSPs and cloud users face a lot of challenges. Challenges are to provide (1) energy efficiency (2) QoS (3) SLA compliance (4) load balancing (5) security (6) traffic management, and (7) cost-effectiveness. Energy efficiency is a major challenge as data centers consume tremendous power and release GHGs, deteriorating the environment. This survey presents a taxonomy of energy-aware optimizations, dynamic power management methods. It is justified that heuristics-based optimization (scheduling) is a generic solution for achieving energy efficiency in CC.

This survey shows that (1) probabilistic algorithms specifically genetic algorithms (PSO, ACO) and bin packing-based algorithms (best fit, EFT, WFD) are the most extensively used techniques for reducing energy consumption. (2) DVFS is the commonly used method for power saving. (3) CloudSim is the widely adopted simulator for evaluation and validation in contrast to real data center implementation. (4) Benchmark programs and real-world traces are the commonly used sources of data. (5) Majorly used parameters are resource utilization, number of cores, node utilization, number of servers, and CPU utilization in contrast to QoS as per SLA compliance. (6) Widely focused GC metrics are GHG emission and PUE which tells the environmental friendliness of a data center.

Many factors such as network characteristics, communication cost, overhead, energy consumption by system components like the disk, main memory were ignored in past research studies. There is a need to design scheduling algorithms which consider the energy consumed by these factors and also work for heterogeneous workloads. Paper also classifies the results into (1) objective results (numerically specified) (2) subjective results (no discrete values). Most of the previous research contributions have objective results. The rapid increase in objective work in this domain indicates its dynamism. The paper concludes by a comprehensive scientometric analysis based on publications while considering bibliometric parameters for analysis. The outcome of this can serve as a direction for future research contribution. Future work will concentrate on the implementation of techniques for energy efficiency in CC and also on investigating the use of scheduling techniques in newly emerged fog and edge environment.