Abstract
The importance of evaluating performance of cloud systems has been increasing with the rapid growing market demands for cloud computing. However, the performance testers often have to go through the hassle of tedious manual operations when interacting with the cloud. A cloud performance evaluation framework is designed for both broad cloud support and good workload extensibility, which provides an automatic interface to monitor the capability and scalability of Infrastructure-as-a-Service cloud systems. Cloud API modules are implemented for Amazon EC2 service and OpenStack. It can achieve flexible control workflows for multiple of different workloads and user customization to test scenarios. With several built-in workloads and metric aggregation methods, a series of tests is performed on our private clouds to compare the performance and scalability from multiple aspects. A methodology is also proposed to build a cost-performance model to better understand and analyze the efficiency of different types of cloud systems. Based on the results of the experiments, the model indicates a polynomial relation between performance per instance and the overall cost.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Bhaskar Prasad, R., Admela, J., Dimitrios, K., Yves, G.: Architectural Requirements for Cloud Computing Systems: An Enterprise Cloud Approach. J. Grid Comput. 9(1), 3–26 (2011)
Thomas, R., Geoff, C., et al.: Grid and cloud computing: opportunities for integration with the next generation network. J. Grid Comput. 7(3), 375–393 (2009)
Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., et al.: A view of cloud computing. Commun. ACM 53(4), 50–58 (2010)
Mell, P., Grance, T.: The nist definition of cloud computing. NIST Spec. Publ. 800(145), 7 (2011)
Wang, L., Von Laszewski, G., Younge, A., He, X., Kunze, M., Tao, J., Fu, C.: Cloud computing: a perspective study. New Gener. Comput. 28(2), 137–146 (2010)
Costa, P., Migliavacca, M., Pietzuch, P., Wolf, A.L.: Naas: network-as-a-service in the cloud. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services, Hot-ICE, vol. 12, pp. 1-1. USENIX, Berkeley (2012)
Shiraz, M., Gani, A., Azra, S., Khan, S., Ahmad, R.W.: Energy efficient computational offloading framework for mobile cloud computing. J. Grid Comput. 13(1), 1–18 (2015)
Yangui, S., Marshall, I.J., Laisne, J.P., Tata, S.: CompatibleOne: the open source cloud broker. J. Grid Comput. 12(1), 93–109 (2014)
van Vliet, J., Paganelli, F.: Programming Amazon EC2. O’Reilly Media, Sebastopol (2011)
Krishnan, S.: Programming Microsoft Azure. O’Reilly Media, Sebastopol (2010)
Hp helion public cloud. http://www.hpcloud.com/
Sabharwal, N., Shankar, R.: Apache CloudStack Cloud Computing. Packt Publishing, Birmingham (2013)
Toraldo, G.: OpenNebula3 Cloud Computing. Packt Publishing, Birmingham (2012)
Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: 2009 9th IEEE International Symposium on Cluster Computing and the Grid, CCGRID’09, IEEE, pp. 124–131. IEEE, Piscataway (2009)
Jackson, K., Bunch, C.: OpenStack Cloud Computing Cookbook, 2nd edn. Packt Publishing, Birmingham (2012)
Avetisyan, A.I., Campbell, R., Gupta, I., Heath, M.T., Ko, S.Y., Ganger, G.R., Kozuch, M.A., O’Hallaron, D., Kunze, M., Kwan, T.T., et al.: Open cirrus: a global cloud computing testbed. Computer 43(4), 35–43 (2010)
Keqin, L.: Optimal load distribution for multiple heterogeneous blade servers in a cloud computing environment. J. Grid Comput. 11(1), 27–46 (2013)
Ge, X., Qi, Z., Chen, K., Duan, J., Dong, Z.: Loosely-coupled benchmark framework automates performance modeling on iaas clouds. In: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, UCC 2014. IEEE, Piscataway (2014)
Casazza, J.P., Greenfield, M., Shi, K.: Redefining server performance characterization for virtualization benchmarking. Intel Technol. J. 3, 10 (2006)
Makhija, V., Herndon, B., Smith, P., Roderick, L., Zamost, E., Anderson, J.: Vmmark: a scalable benchmark for virtualized systems. VMware Inc, CA, Tech. Rep. VMware-TR-2006-002 (2006)
Muñoz, V.M., Ramo, A.C., Diaz, R.G., Tsaregorodtsev, A.: Cloud governance by a credit model with DIRAC. In: Proceedings of the 4th International Conference on Cloud Computing and Services Science, pp. 679–686 (2014)
Zheng, X., Martin, P., Brohman, K., Xu, L.D.: Cloud service negotiation in internet of things environment: A mixed approach. IEEE Trans. Ind. Inf. 10(2), 1506–1515 (2014)
Tao, F., Laili, Y., Xu, L., Zhang, L.: Fc-paco-rm: A parallel method for service composition optimal-selection in cloud manufacturing system. IEEE Trans. Ind. Inf. 9(4), 2023–2033 (2013)
Wang, L., Zhan, J., Luo, C., Zhu, Y., Yang, Q., He, Y., Gao, W., Jia, Z., Shi, Y., Zhang, S., Zheng, C., Lu, G., Zhan, K., Li, X., Qiu, B.: Bigdatabench: A big data benchmark suite from internet services. In: Proceedings of the 20th IEEE International Symposium on High Performance Computer Architecture, HPCA’2014. Piscataway, NJ, USA: IEEE, pp. 488–499 (2014)
Zheng, X., Martin, P., Brohman, K., Xu, L.D.: Cloudqual: A quality model for cloud services. IEEE Trans. Ind. Inf. 10(2), 1527–1536 (2014)
Tao, F., Zuo, Y., Xu, L.D., Zhang, L.: Iot-based intelligent perception and access of manufacturing resource toward cloud manufacturing. IEEE Trans. Ind. Inf. 10(2), 1547–1557 (2014)
Muñoz, V.M., Ramo, A.C., Albor, V.F., Diaz, R.G., Arévalo, G.M.: Rafhyc: an architecture for constructing resilient services on federated hybrid clouds. J. Grid Comput. 11(4), 753–770 (2013)
Cloudharmony: simplify the comparison of cloud services, Laguna Beach, CA, USA, http://cloudharmony.com/
Li, A., Yang, X., Kandula, S., Zhang, M.: Cloudcmp: comparingpublic cloud providers. In: Proceedings of the 2010 conferenceon Internet measurement conference, ACM. NewYork, NY, USA: ACM, pp. 1–14 (2010)
Novakovic, D., Vasic, N., Novakovic, S., Kostic, D., Bianchini, R.: Deepdive: Transparently identifying and managing performance interference in virtualized environments. In: Proceedings of USENIX ATC’13: 2013 USENIX Annual Technical Conference, ATC’13. Berkeley, CA, USA: USENIX, pp. 219–230 (2013)
Ferdman, M., Adileh, A., Kocberber, O., Volos, S., Alisafaee, M., Jevdjic, D., Kaynak, C., Popescu, A.D., Ailamaki, A., Falsafi, B.: Clearing the clouds: a study of emerging scale-out workloads on modern hardware. In: ACM SIGARCH Computer Architecture News, vol. 40, no. 1, ACM. NewYork, NY, USA: ACM, pp. 37–48 (2012)
El-Refaey, M.A., Rizkaa, M.A.: Cloudgauge: a dynamic cloud and virtualization benchmarking suite. In: 2010 19th IEEE International Workshop on Enabling Technologies: Infrastructures for Collaborative Enterprises, IEEE. Piscataway, NJ, USA: IEEE, pp. 66–75 (2010)
Silva, M., Hines, M.R., Gallo, D., Liu, Q., Ryu, K.D., da Silva, D.: Cloudbench: Experiment automation for cloud environments. In: Proceedings of the 2013 IEEE International Conference on Cloud Engineering, IC2E’13, IEEE. Piscataway, NJ, USA: IEEE, pp. 302–311 (2013)
Openstack Rally project, https://wiki.openstack.org/wiki/Rally
Fuel: Openstack deployment and management,” Mountain View, CA, USA, http://docs.mirantis.com/openstack/fuel/fuel-5.1/planning-guide.html
Acknowledgments
This work was supported in part by National Nature Science Foundation of China (61073151, 61572195), Shanghai Agriculture Science Program (2015.3-2), and Shanghai Key Laboratory of Trustworthy Computing Open Project Fund (07dz22304201608).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Mao, H., Qi, Z., Duan, J. et al. Cost-Performance Modeling with Automated Benchmarking on Elastic Computing Clouds. J Grid Computing 15, 557–572 (2017). https://doi.org/10.1007/s10723-017-9412-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10723-017-9412-4