Abstract
Heuristic approaches require fixed knowledge of how resource allocation should be carried out, and this can be limiting when managing variable cloud workloads. Solutions based on Reinforcement Learning (RL) have been presented to manage cloud infrastructure, however, these tend to be centralized and suffer in their ability to maintain Quality of Service (QoS) for data centres with thousands of nodes. To address this, we propose a reinforcement learning management policy, which can run decentralized, and achieve fast convergence towards efficient resource allocation, resulting in lower SLA violations compared to centralized architectures. To address some of the common challenges in applying RL to cloud resource management, such as slow learning and state/action management, we use parallel learning and reduction of the state/action space. We apply a decision making approach to optimize the migration of a VM and choose a target node to host the VM in such a way that brings response time within SLA level. We have also demonstrate unique, multi-level reinforcement learning cooperation, that further reduces SLA violations. We use simulation to evaluate and demonstrate our proposal in practice, and compare the results obtained with an established heuristic, demonstrating significant improvement to SLA violations and higher scalability.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Data Availability
The data traces used as input for this paper are part of the DCSim [57] simulator, and are available from Github: https://github.com/digs-uwo/dcsim/tree/master/traces. Most of the data generated by this research is contained in the results section of this paper, and the full datasets generated are available from the corresponding author on request.
References
Hpe proliant (2016). https://www.hpe.com
Vmware. (2016). http://www.vmware.com/
Ahmad, R. W., Gani, A., Hamid, S. H. A., Shiraz, M., Yousafzai, A., Xia, F.: A survey on virtual machine migration and server consolidation frameworks for cloud data centers. J. Netw. Comput. Appl. 52, 11–25 (2015). https://doi.org/10.1016/j.jnca.2015.02.002
Aldhalaan, A., Menascé, D. A.: Autonomic Allocation of Communicating Virtual Machines in Hierarchical Cloud Data Centers. In: 2014 International Conference On Cloud and Autonomic Computing (ICCAC), pp. 161–171. 2014 International Conference On Cloud and Autonomic Computing (2014) (2014)
Arabnejad, H., Pahl, C., Jamshidi, P., Estrada, G.: A Comparison of Reinforcement Learning Techniques for Fuzzy Cloud Auto-Scaling. In: 2017 17Th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 64–73. https://doi.org/10.1109/CCGRID.2017.15 (2017)
Barrett, E., Howley, E., Duggan, J.: Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurr. Comput. Pract. Exper. 25(12), 1656–1674 (2013). https://doi.org/10.1002/cpe.2864
Beloglazov, A., Buyya, R.: Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers. Concurr. Comput. Pract. Exper. 24, 1397–1420 (2012)
Bibal Benifa, J. V., Dejey, D.: Rlpas: Reinforcement learning-based proactive auto-scaler for resource provisioning in cloud environment. Mobile Netw. Appl. 24(4), 1348–1363 (2019). https://doi.org/10.1007/s11036-018-0996-0
Bitsakos, C., Konstantinou, I., Koziris, N.: Derp: a Deep Reinforcement Learning Cloud System for Elastic Resource Provisioning. In: 2018 IEEE International Conference on Cloud Computing Technology and Science (Cloudcom), pp. 21–29. https://doi.org/10.1109/CloudCom2018.2018.00020 (2018)
Bu, X., Rao, J., Xu, C. Z.: Coordinated self-configuration of virtual machines and appliances using a model-free learning approach. IEEE Trans. Parallel Distrib. Syst. 24(4), 681–690 (2013). https://doi.org/10.1109/TPDS.2012.174
Calcavecchia, N. M., Caprarescu, B. A., Di Nitto, E., Dubois, D. J., Petcu, D.: Depas: a decentralized probabilistic algorithm for auto-scaling. Computing 94(8), 701–730 (2012)
Chen, Z., Hu, J., Min, G.: Learning-Based Resource Allocation in Cloud Data Center Using Advantage Actor-Critic. In: ICC 2019 - 2019 IEEE International Conference on Communications (2019), pp. 1–6. https://doi.org/10.1109/ICC.2019.8761309 (2019)
Chowdhury, M. R., Mahmud, M. R., Rahman, R. M.: Implementation and performance analysis of various vm placement strategies in cloudsim. J. Cloud Comput. 4(1), 20 (2015). https://doi.org/10.1186/s13677-015-0045-5
Citrix: Xen. (2016). http://www.xenserver.org
Duggan, M., Flesk, K., Duggan, J., Howley, E.: Barrett e.: a reinforcement learning approach for dynamic selection of virtual machines in cloud data centres. https://doi.org/10.1109/INTECH.2016.7845053(2016)
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., Hester, T.: Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning. https://doi.org/10.1007/s10994-021-05961-4 (2021)
Gahlawat, M., Sharma, P.: Survey of Virtual Machine Placement in Federated Clouds. In: 2014 IEEE International Advance Computing Conference (IACC), pp. 735–738. https://doi.org/10.1109/IAdCC.2014.6779415 (2014)
Ghanbari, H., Simmons, B., Litoiu, M., Barna, C., Iszlai, G.: Optimal autoscaling in a iaas cloud. In: Proceedings of the 9th International Conference on Autonomic Computing, ICAC ’12, pp 173–178. Association for Computing Machinery, New York (2012), https://doi.org/10.1145/2371536.2371567
Ghobaei-Arani, M., Jabbehdari, S., Pourmina, M. A.: An autonomic resource provisioning approach for service-based cloud applications: a hybrid approach. Fut. Gener. Comput. Syst. 78, 191–210 (2018). https://doi.org/10.1016/j.future.2017.02.022
Gholipour, N., Arianyan, E., Buyya, R.: A novel energy-aware resource management technique using joint vm and container consolidation approach for green computing in cloud data centers. Simul. Model. Pract. Theory 104, 102127 (2020). https://doi.org/10.1016/j.simpat.2020.102127
Guo, W., Tian, W., Ye, Y., Xu, L., Wu, K.: Cloud resource scheduling with deep reinforcement learning and imitation learning. IEEE Internet Things J. 8(5), 3576–3586 (2021). https://doi.org/10.1109/JIOT.2020.3025015
Gupta, M. K., Amgoth, T.: Resource-aware virtual machine placement algorithm for iaas cloud. J. Supercomput. 74(1), 122–140 (2018). https://doi.org/10.1007/s11227-017-2112-9
Hummaida, A. R., Paton, N. W., Sakellariou, R.: Adaptation in cloud resource configuration: a survey. J. Cloud Comput. 5(1), 1–16 (2016)
Hummaida, A. R., Paton, N. W., Sakellariou, R.: Shdf - a Scalable Hierarchical Distributed Framework for Data Centre Management. In: 2017 16Th International Symposium on Parallel and Distributed Computing (ISPDC), pp. 102–111. 16Th International Symposium on Parallel and Distributed Computing (2017). https://doi.org/10.1109/ISPDC.2017.15 (2017)
Jamshidi, P., Sharifloo, A. M., Pahl, C., Metzger, A., Estrada, G.: Self-Learning Cloud Controllers: Fuzzy Q-Learning for Knowledge Evolution. In: 2015 International Conference on Cloud and Autonomic Computing, pp. 208–211. https://doi.org/10.1109/ICCAC.2015.35 (2015)
Jangiti, S., Sriram, V.S.: Scalable and direct vector bin-packing heuristic based on residual resource ratios for virtual machine placement in cloud data centers. Comput. Electr. Eng. 68, 44–61 (2018). https://doi.org/10.1016/j.compeleceng.2018.03.029
Jauro, F., Chiroma, H., Gital, A. Y., Almutairi, M., Abdulhamid, S. M., Abawajy, J. H.: Deep learning architectures in emerging cloud computing architectures: Recent development, challenges and next research trend. Appl. Soft Comput. 106582, 96 (2020). https://doi.org/10.1016/j.asoc.2020.106582
Jin, Y., Bouzid, M., Kostadinov, D., Aghasaryan, A.: Resource management of cloud-enabled systems using model-free reinforcement learning. Ann. Telecommun. 74(9), 625–636 (2019). https://doi.org/10.1007/s12243-019-00720-y
John, I., Sreekantan, A., Bhatnagar, S.: Efficient adaptive resource provisioning for cloud applications using reinforcement learning. In: 2019 IEEE 4Th International Workshops on Foundations and Applications of Self* Systems (FAS*W), pp. 271–272. https://doi.org/10.1109/FAS-W.2019.00077 (2019)
Kardani-Moghaddam, S., Buyya, R., Ramamohanarao, K.: Adrl: a hybrid anomaly-aware deep reinforcement learning-based resource scaling in clouds. IEEE Trans Parallel Distrib Syst 32 (3), 514–526 (2021). https://doi.org/10.1109/TPDS.2020.3025914
Keller, G., Tighe, M., Lutfiyya, H., Bauer, M.: A Hierarchical, Topology-Aware Approach to Dynamic Data Centre Management. In: Network Operations and Management Symposium (NOMS), pp. 1 –7. Network Operations and Management Symposium (2014) (2014)
Khan, T., Tian, W., Buyya, R.: Machine learning (ml)-centric resource management in cloud computing: A review and future directions (2021)
Kim, S., Choi, Y.R.: Constraint-aware vm placement in heterogeneous computing clusters. Clust. Comput. 23(1), 71–85 (2020). https://doi.org/10.1007/s10586-019-02966-6
Lebre, A., Pastor, J., Simonet, A., Südholt, M.: Putting the next 500 vm placement algorithms to the acid test: The infrastructure provider viewpoint. IEEE Trans. Parallel Distrib. Syst. 30(1), 204–217 (2019). https://doi.org/10.1109/TPDS.2018.2855158
Liu, N., Li, Z., Xu, J., Xu, Z., Lin, S., Qiu, Q., Tang, J., Wang, Y.: A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning. In: 2017 IEEE 37Th International Conference on Distributed Computing Systems (ICDCS), pp. 372–382. https://doi.org/10.1109/ICDCS.2017.123 (2017)
Lolos, K., Konstantinou, I., Kantere, V., Koziris, N.: Elastic Management of Cloud Applications Using Adaptive Reinforcement Learning. In: 2017 IEEE International Conference on Big Data (Big Data), pp. 203–212. https://doi.org/10.1109/BigData.2017.8257928 (2017)
Masdari, M., Zangakani, M.: Green cloud computing using proactive virtual machine placement: Challenges and issues. J. Grid Comput. 18(4), 727–759 (2020). https://doi.org/10.1007/s10723-019-09489-9
Matignon, L., Laurent, G. J., Fort-piat, N. L.: Improving Reinforcement Learning Speed for Robot Control. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3172–3177. https://doi.org/10.1109/IROS.2006.282341 (2006)
Maurer, M., Brandic, I., Sakellariou, R.: Adaptive resource configuration for cloud infrastructure management. Futur. Gener. Comput. Syst. 29(2), 472–487 (2013)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236
Moreno-Vozmediano, R., Montero, R. S., Huedo, E., Llorente, I. M.: Efficient resource provisioning for elastic cloud services based on machine learning techniques. J. Cloud Comput. 8(1), 5 (2019). https://doi.org/10.1186/s13677-019-0128-9
Muller-Merbach, H.: Heuristics and their design: a survey. Eur. J. Oper. Res. 8(1), 1–23 (1981). https://ideas.repec.org/a/eee/ejores/v8y1981i1p1-23.html
Nouri, S. M. R., Li, H., Venugopal, S., Guo, W., He, M., Tian, W.: Autonomic decentralized elasticity based on a reinforcement learning controller for cloud applications. Fut. Gener. Comput. Syst. 94, 765–780 (2019). https://doi.org/10.1016/j.future.2018.11.049
Pantazoglou, M., Tzortzakis, G., Delis, A.: Decentralized and energy-efficient workload management in enterprise clouds. IEEE Trans. Cloud Comput. 4(2), 196–209 (2016)
Pietri, I., Sakellariou, R.: Mapping virtual machines onto physical machines in cloud computing: A survey. ACM Comput. Surv 49(3). https://doi.org/10.1145/2983575 (2016)
Rao, J., Bu, X., Xu, C. Z., Wang, K.: A Distributed Self-Learning Approach for Elastic Provisioning of Virtualized Cloud Resources. In: 2011 IEEE 19Th Annual International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, pp. 45–54. https://doi.org/10.1109/MASCOTS.2011.47 (2011)
Ren, H., Wang, Y., Xu, C., Chen, X.: Smig-rl: An evolutionary migration framework for cloud services based on deep reinforcement learning. ACM Trans. Internet Technol 20(4). https://doi.org/10.1145/3414840 (2020)
Sedaghat, M., Hernández-Rodriguez, F., Elmroth, E., Girdzijauskas, S.: Divide the Task, Multiply the Outcome: Cooperative Vm Consolidation. In: IEEE International Conference on Cloud Computing Technology and Science, pp. 300–305. IEEE International Conference on Cloud Computing Technology and Science, IEEE, Washington (2014)
Shaw, R., Howley, E., Barrett, E.: Applying reinforcement learning towards automating energy efficient virtual machine consolidation in cloud data centers. Inf. Syst., 101722. https://doi.org/10.1016/j.is.2021.101722 (2021)
Silva Filho, M.C., Monteiro, C.C., Inácio, P.R., Freire, M.M.: Approaches for optimizing virtual machine placement and migration in cloud environments: a survey. J. Parallel Distrib. Comput. 111, 222–250 (2018). https://doi.org/10.1016/j.jpdc.2017.08.010
Sina, M., Dehghan, M., Rahmani, A. M.: Car-plive: Cloud-assisted reinforcement learning based p2p live video streaming: a hybrid approach. Multimed. Tools Appl. 78(23), 34095–34127 (2019). https://doi.org/10.1007/s11042-019-08102-1
Sniezynski, B., Nawrocki, P., Wilk, M., Jarzab, M., Zielinski, K.: Vm reservation plan adaptation using machine learning in cloud computing. J. Grid Comput. 17(4), 797–812 (2019). https://doi.org/10.1007/s10723-019-09487-x
Song, B., Hassan, M., Huh, E.N.: A Novel Heuristic-Based Task Selection and Allocation Framework in Dynamic Collaborative Cloud Service Platform. In: 2010 IEEE Second International Conference on Cloud Computing Technology and Science, pp. 360–367. https://doi.org/10.1109/CloudCom.2010.53 (2010)
Sutton, R. S., Barto, A. G.: Reinforcement Learning: an Introduction, vol. 1. MIT press, Cambridge (1998)
Thanh Binh, H. T., Phi Le, N., Minh, N. B., Thu Hai, T., Minh, N. Q., Bao Son, D.: A Reinforcement Learning Algorithm for Resource Provisioning in Mobile Edge Computing Network. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. https://doi.org/10.1109/IJCNN48605.2020.9206947 (2020)
Tighe, M., Keller, G., Bauer, M.: Lutfiyya: a distributed approach to dynamic vm management. In: Proceedings of the 9th International Conference on Network and Service Management, pp. 166 to 170. Proceedings of the 9th International Conference on Network and Service Management (2013)
Tighe, M., Keller, G., Bauer, M., Lutfiyya, H.: Dcsim: a Data Centre Simulation Tool for Evaluating Dynamic Virtualized Resource Management. In: Network and Service Management (Cnsm), 2012 8Th International Conference and 2012 Workshop on Systems Virtualiztion Management (Svm), pp. 385–392. Network and Service Management (Cnsm), 2012 8Th International Conference and 2012 Workshop on Systems Virtualiztion Management (Svm) (2012)
Walsh, W., Tesauro, G., Kephart, J., Das, R.: Utility Functions in Autonomic Systems. In: 2004. Proceedings. International Conference on Autonomic Computing, pp. 70–77. https://doi.org/10.1109/ICAC.2004.1301349 (2004)
Watkins, C. J. C. H.: Learning from Delayed Rewards. In: Ph.D. Thesis (1989)
Witanto, J. N., Lim, H., Atiquzzaman, M.: Adaptive selection of dynamic vm consolidation algorithm using neural network for cloud resource management. Futur. Gener. Comput. Syst. 87, 35–42 (2018). https://doi.org/10.1016/j.future.2018.04.075
Wu, Y., Tang, M., Fraser, W.: A Simulated Annealing Algorithm for Energy Efficient Virtual Machine Placement. In: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1245–1250. https://doi.org/10.1109/ICSMC.2012.6377903 (2012)
Wuhib, F., Stadler, R., Spreitzer, M.: Dynamic resource allocation with management objectives: implementation for an openstack cloud. IEEE Trans. Netw. Serv. Manag. 9(2), 213–225 (2012)
Xu, H., Liu, Y., Wei, W., Xue, Y.: Migration cost and energy-aware virtual machine consolidation under cloud environments considering remaining runtime. Int. J. Parallel Prog. 47(3), 481–501 (2019). https://doi.org/10.1007/s10766-018-00622-x
Yadav, M. P., Rohit Yadav, D.K.: Resource provisioning through machine learning in cloud services. Arabian Journal for Science and Engineering. https://doi.org/10.1007/s13369-021-05864-5 (2021)
Yadav, R., Zhang, W., Li, K., Liu, C., Shafiq, M., Karn, N. K.: An adaptive heuristic for managing energy consumption and overloaded hosts in a cloud data center. Wirel. Netw 26(3), 1905–1919 (2020). https://doi.org/10.1007/s11276-018-1874-1
Ying, C., Li, B., Ke, X., Guo, L.: Raven: Scheduling virtual machine migration during datacenter upgrades with reinforcement learning. Mobile Networks and Applications. https://doi.org/10.1007/s11036-020-01632-1 (2020)
Zolfaghari, R., Sahafi, A., Rahmani, A. M., Rezaei, R.: Application of virtual machine consolidation in cloud computing systems. Sustain. Comput. Inf. Syst. 30, 100524 (2021). https://doi.org/10.1016/j.suscom.2021.100524
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hummaida, A.R., Paton, N.W. & Sakellariou, R. Scalable Virtual Machine Migration using Reinforcement Learning. J Grid Computing 20, 15 (2022). https://doi.org/10.1007/s10723-022-09603-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10723-022-09603-4