Abstract
Novel applications will require extending traditional cloud computing infrastructure with compute resources deployed close to the end user. Edge and fog computing tightly integrated with carrier networks can fulfill this demand. The emphasis is on integration: the rigorous delay constraints, ensuring reliability on the distributed, remote compute nodes, and the sheer scale of the system altogether call for a powerful resource provisioning platform that offers the applications the best of the underlying infrastructure. We therefore propose Kubernetes-edge-scheduler that provides high reliability for applications in the edge, while provisioning less than 10% of resources for this purpose, and at the same time, it guarantees compliance with the latency requirements that end users expect. We present a novel topology clustering method that considers application latency requirements, and enables scheduling applications even on a worldwide scale of edge clusters. We demonstrate that in a potential use case, a distributed stream analytics application, our orchestration system can reduce the job completion time to 40% of the baseline provided by the default Kubernetes scheduler.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Aral, A., Brandic, I., Uriarte, R.B., De Nicola, R., Scoca, V.: Addressing application latency requirements through edge scheduling. J. Grid Comput., 17. https://doi.org/10.1007/s10723-019-09493-z(2019)
Bose, S.K., Brock, S., Skeoch, R., Rao, S.: Cloudspider: Combining replication with scheduling for optimizing live migration of virtual machines across wide area networks. In: Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (2011)
Chima Ogbuachi, M., Gore, C., Reale, A., Suskovics, P., Kovács, B.: Context-Aware K8S Scheduler for Real Time Distributed 5G Edge Computing Applications. In: 2019 International Conference on Software, Telecommunications and Computer Networks (SoftCOM), pp. 1–6. https://doi.org/10.23919/SOFTCOM.2019.8903766 (2019)
Fan, J., Jiang, M., Rottenstreich, O., Zhao, Y., Guan, T., Ramesh, R., Das, S., Qiao, C.: A framework for provisioning availability of nfv in data center networks. IEEE J. Sel. Areas Commun. 36(10), 2246–2259 (2018). https://doi.org/10.1109/JSAC.2018.2869960
Farris, I., Taleb, T., Flinck, H., Iera, A.: Providing ultra-short latency to user-centric 5g applications at the mobile network edge. Trans. Emerging Telecommun. Technol. 29(4). https://doi.org/10.1002/ett.3169 (2018)
Goldpinger: Debugging tool for Kubernetes. https://github.com/bloomberg/goldpinger. Accessed on: 21 March 2020
Haja, D., Szabo, M., Szalay, M., Nagy, A., Kern, A., Toka, L., Sonkoly, B.: How to Orchestrate a Distributed OpenStack. In: IEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp. 293–298. https://doi.org/10.1109/INFCOMW.2018.8407014 (2018)
Haja, D., Szalay, M., Sonkoly, B., Pongracz, G., Toka, L.: Sharpening kubernetes for the edge. In: Proceedings of the ACM SIGCOMM 2019 Conference Posters and Demos, SIGCOMM Posters and Demos ’19. https://doi.org/10.1145/3342280.3342335, pp 136–137. Association for Computing Machinery, New York (2019)
Huedo, E., Montero, R.S., Moreno-Vozmediano, R., Vázquez, C., Holer, V., Llorente, I.M.: Opportunistic deployment of distributed edge clouds for latency-critical applications. J. Grid Comput., 19. https://doi.org/10.1007/s10723-021-09545-3 (2021)
Jain, N., Chana, I.: Energy-aware Virtual Machine Migration for Cloud Computing - A Firefly Optimization Approach. J. Grid Comput., 14. https://doi.org/10.1007/s10723-016-9364-0(2016)
Javed, A., Robert, J., Heljanko, K., Främling, K.: IoTEF: A Federated Edge-Cloud Architecture for Fault-Tolerant IoT Applications. J. Grid Comput., 18. https://doi.org/10.1007/s10723-019-09498-8 (2020)
K3s: Lightweight Kubernetes. https://k3s.io/. Accessed on: 21 March 2020
Kanizo, Y., Rottenstreich, O., Segall, I., Yallouz, J.: Optimizing virtual backup allocation for middleboxes. IEEE/ACM Trans. Netw. 25(5), 2759–2772 (2017). https://doi.org/10.1109/TNET.2017.2703080
Kosińska, J., Zielinski, K.: Autonomic management framework for cloud-native applications. J. Grid Comput., 18. https://doi.org/10.1007/s10723-020-09532-0 (2020)
KubeEdge: A Kubernetes Native Edge Computing Framework. https://kubeedge.io. Accessed on: 21 March 2020
Kubernetes Cluster Federation. https://github.com/kubernetes-sigs/kubefed. Accessed on: 21 March 2020
Kubernetes: Production-grade Container Orchestration. https://kubernetes.io. Accessed on: 21. March 2020
kubernetes-edge-scheduler. https://github.com/davidhaja/kubernetes-edge-scheduler. Accessed on: 21 March 2020
MicroK8s: Lightweight upstream K8s. https://microk8s.io Accessed on: 21 March 2020
Nakanishi, K., Suzuki, F., Ohzahata, S., Yamamoto, R., Kato, T.: A Container-Based Content Delivery Method for Edge Cloud over Wide Area Network. In: 2020 International Conference on Information Networking (ICOIN), pp. 568–573. https://doi.org/10.1109/ICOIN48656.2020.9016481 (2020)
Németh, B., Szalay, M., Dóka, J., Rost, M., Schmid, S., Toka, L., Sonkoly, B.: Fast and Efficient Network Service Embedding Method with Adaptive Offloading to the Edge. In: IEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp 178–183 (2018). https://doi.org/10.1109/INFCOMW.2018.8406882
Santos, J., Wauters, T., Volckaert, B., De Turck, F.: Towards Network-Aware Resource Provisioning in Kubernetes for Fog Computing Applications. In: 2019 IEEE Conference on Network Softwarization (Netsoft), pp. 351–359. https://doi.org/10.1109/NETSOFT.2019.8806671 (2019)
Shang, X., Huang, Y., Liu, Z., Yang, Y.: Reducing the Service Function Chain Backup Cost over the Edge and Cloud by a Self-Adapting Scheme. In: IEEE INFOCOM -IEEE Conference on Computer Communications (2020)
Szalay, M., Mátray, P., Toka, L.: Minimizing State Access Delay for Cloud-Native Network Functions. In: 2019 IEEE 8th International Conference on Cloud Networking (Cloudnet), pp. 1–6. https://doi.org/10.1109/CloudNet47604.2019.9064048 (2019)
Szalay, M., Mátray, P., Toka, L.: State management for cloud-native applications. Electronics 10(4). https://doi.org/10.3390/electronics10040423. https://www.mdpi.com/2079-9292/10/4/423 (2021)
Szalay, M., Nagy, M., Géhberger, D., Kiss, Z., Mátray, P., Németh, F., Pongrácz, G., Rétvári, G., Toka, L.: Industrial-Scale Stateless Network Functions. In: 2019 IEEE 12Th International Conference on Cloud Computing (CLOUD), pp. 383–390. https://doi.org/10.1109/CLOUD.2019.00068 (2019)
Toka, L., Haja, D., Kőrösi, A., Sonkoly, B.: Resource Provisioning for Highly Reliable and Ultra-Responsive Edge Applications. In: 2019 IEEE 8Th International Conference on Cloud Networking (Cloudnet), pp. 1–6. https://doi.org/10.1109/CloudNet47604.2019.9064131 (2019)
Yala, L., Frangoudis, P.A., Ksentini, A.: Latency and Availability Driven VNF Placement in a MEC-NFV Environment. In: IEEE Global Communications Conference (GLOBECOM) (2018)
Yao, J., Ansari, N.: Reliability-Aware Fog Resource Provisioning for Deadline-Driven IoT Services. In: IEEE Global Communications Conference (GLOBECOM) (2018)
Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: Cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud’10, p 10. USENIX Association, USA (2010)
Zhang, J., Wang, Z., Peng, C., Zhang, L., Huang, T., Liu, Y.: Raba: Resource-Aware Backup Allocation for a Chain of Virtual Network Functions. In: IEEE INFOCOM 2019-IEEE Conference on Computer Communications (2019)
Acknowledgements
Projects no. 135074 and 2019-2.1.13-TÉT-IN-2020-00021 have been implemented with the support provided by the Ministry of Innovation and Technology of Hungary from the National Research, Development and Innovation Fund, financed under the FK20 and 2019-2.1.13-TÉT-IN funding schemes.
Funding
Open access funding provided by Budapest University of Technology and Economics.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Lemma 5
In case of a fix number of vertices (|V | = n) and a fix diameter (d), with the increase of the number of buds (b), the optimal solution (OPT) monotonically increases.
Proof
Referring on the proof of Lemma 2, one can see that the number of placeholders is directly proportional to the number of buds. □
Lemma 6
In case of a fix number of vertices (|V | = n) and a fix number of buds (b), with the increasing diameter of the graph d, the number of placeholders, given by our heuristic algorithm (HEUR) monotonically increases.
Proof
Let us induce a path graph G′ from a path in G whose length equals to d. We can give a sequence of Pod requests, for which the amount of placeholders provisioned by our heuristic solution would be equal to HEUR = d. Since G′ has k = d + 1 vertices, the HEUR = d = k − 1 solution is the worst solution that our algorithm can give on G′ (Lemma 3). Regarding this, one can see that with the increase of a graph’s diameter, the number of placeholders, given by our heuristic algorithm, monotonically increases. □
Lemma 7
We deduce the possible number of buds in simple, connected graphs with a given diameter.
-
1.
If d = 1, then b = 0;
-
2.
If d = 2, then 0 ≤ b ≤ 1;
-
3.
If \(3 \leq d \leq \frac {n}{2}\), then \(0 \leq b \leq \frac {n}{2}\);
-
4.
If \(d=\frac {n}{2}+k, k > 0 (\frac {n}{2}+1 \leq d \leq n-3)\), then \(0 \leq b \leq (\frac {n}{2})-k+1\);
-
5.
If d = n − 2, then 1 ≤ b ≤ 3;
-
6.
If d = n − 1, then b = 2.
Proof
The indices of the following proofs refer to the indices of cases listed in the lemma.
-
1.
Only complete graphs (from the class of connected, simple graphs) have diameter 1. Complete graphs do not have any leaf vertices, therefore they do not have buds either.
-
2.
The diameter of a star graph, with one central node is 2. This graph has exactly 1 bud node (the central node), since all other nodes are leaves. We can construct several different graphs that have diameter d = 2 and 0 buds. One trivial example is a graph that is constructed from a complete graph by deleting a single edge. There cannot be more buds in graphs with d = 2, since the smallest combination of vertices and edges where b = 2 is a leaf-bud-bud-leaf subgraph that already has d = 3.
-
3.
Let us create a cycle with n vertices. This cycle has \(d=\frac {n}{2}\) and b = 0. We can add extra diagonal edges to this graph so that it will have any diameter value between 3 and \(\frac {n}{2}\) and the number of buds remains 0.
Now let us present the other cases, when the graphs have \(0 < b \leq \frac {n}{2}\). For every diameter value \(3 \leq d \leq \frac {n}{4}\) we can create a cycle C with \(\frac {n}{2}\) vertices and adding extra diagonal edges so that G has \(3 \leq d \leq \frac {n}{4}\). At this point, we can connect the remaining (G − C) \(\frac {n}{2}\) vertices (from “outside of the circle”) to the cycle so the number of buds is \(0 < b \leq \frac {n}{2}\).
We can also construct graphs for which \(\frac {n}{4} < d \leq \frac {n}{2}\) and \(0 < b \leq \frac {n}{2}\). Let us create a path graph with d + 1 vertices. The number of internal nodes (that are neither buds, nor leaves in the initial path graph) is d + 1 − 4 = d − 3. We can add d − 3 extra leaves connected to the internal nodes in order to increase b. To increase further the value of b, we can connect extra bud-leaf node pairs to any of the internal nodes till we reach the desired number of buds, or we consumed all the vertices in the graph. In case we reach the desired number of buds, but there are more unconnected vertices, we can construct a clique from them and connect it to any internal node.
We showcased possible diameter and bud combinations between the upper and lower bounds of both the diameter and the number of buds. We also showed how we can construct graphs that have any d and b values between the defined bounds. Since the maximum number of buds in a graph with n vertices is \(\frac {n}{2}\), there is no other combination that can be constructed regarding the given diameter range.
-
4.
Since the diameter of the graph is greater than \(\frac {n}{2}\), the graph can not be a cycle. Let us construct a path graph with \(\frac {n}{2}+k+1\) vertices (so the diameter equals to \(\frac {n}{2}+k\)). Following on Lemma 7, we can close the two ends of the path graph with two vertices, and connect the remaining (\(\frac {n}{2}-k-3\)) vertices so that the graph will not have any buds. After the induction of the path graph with \(\frac {n}{2}+k+1\) vertices, it has \(\frac {n}{2}+k+1-4=\frac {n}{2}+k-3\) vertices and we have \(\frac {n}{2}-k-1\) unconnected nodes. The maximum number of buds that is achievable in these scenarios is \(\frac {n}{2}-k-1+2=\frac {n}{2}-k+1\), since the path graph already has 2 buds, one at each end.
-
5.
Let us construct a path graph with n − 1 vertices with d = n − 2 and b = 2. The remaining one node can be connected to the graph in three possible ways: i) the graph will have b = 1 bud, if we form a triangle at one end of the path graph (so we connect the last node to a leaf and to the connected bud); ii) the graph will have b = 2 buds, if we connect the last node to two adjacent internal nodes; iii) the graph will have b = 3 buds, if we connect the last node to one of the internal nodes.
-
6.
A path graph with n nodes is the only connected, simple graph that has d = n − 1. This graph must have b = 2 buds, i.e., the second vertex on both ends.
□
Lemma 8
The approximation ratio between our heuristic and the optimal solution for given maximal diameter and minimal number of buds:
-
1.
if d = 1 and b = 0, then HEUR = OPT = 2;
-
2.
if d = n − 3 and b = 0, then HEUR ≤ 3OPT;
-
3.
if d = n − 2 and b = 1, then HEUR ≤ 3OPT;
-
4.
if d = n − 1 and b = 2, then HEUR ≤ 3OPT.
Proof
The indices of the following proofs refer to the indices of cases listed in the lemma.
-
1.
The optimal solution in every complete graph is 2. The anti-affinity requirement hinders to have only one placeholder. Our heuristic algorithm also achieves the value of 2, since after the deployment of the first request there will be one Pod and one placeholder in the system. Let us denote the hosting node of this placeholder with v. Every other Pod whose origin differs from v will be placed in the system without the need of increasing the number of the total placeholders. For the Pod, whose origin host is v, our solution creates the second placeholder, and deploys the Pod on another node. Therefore, with any order of Pod submissions, the heuristic algorithm will deploy only two placeholders.
-
2.
Let us construct a path graph with n − 2 nodes with d = n − 3 and b = 2. Now let us close both ends of this path graph with two triangles made by the last two nodes on both ends and the two unconnected nodes. Therefore, the graph will have b = 0. In this case the optimal solution equals to \(OPT = \frac {d+1}{3} = \frac {n-2}{3}\) (following on Lemma 1). Our heuristic solution will deploy maximum two placeholders in each triangle, so at least one of the Pods (per side) whose origin is in the triangle will benefit from a previously deployed placeholder, i.e., two Pods will share a placeholder on both ends of the graph. From this, we can state that at least two nodes will not have any placeholder on them, so HEUR ≤ n − 2. Therefore, \(HEUR \leq 3 OPT = n-2 \leq 3 \frac {n-2}{3}\).
-
3.
Regarding to Lemma 1, the optimal solution can not be less than \(OPT = \frac {d+1}{3} = \frac {n-1}{3}\). Therefore, even when the heuristic solution achieves the worst solution (HEUR ≤ n − 1), we can state that \(HEUR \leq 3 OPT = n-1 \leq 3 \frac {n-1}{3}\). One can see that our statement HEUR ≤ 3OPT is proven in graphs with d = n − 2 and b = 1.
-
4.
The only connected, simple graph that has d = n − 1 and b = 2 is the path graph with n nodes (see Lemma 7). The Pods’ delay requirements allow only the origin node and its neighbors (see Assumption 1) as their hosting node. Therefore, the number of vertices in each Pod’s radius is 2 or 3. Therefore, the minimum number of sets that cover all nodes, each set containing only nodes that are connected with each other, give the optimal solution for the number of placeholders. Therefore the optimal solution is \(OPT = \lceil \frac {n}{3} \rceil \) in a path graph with n vertices. In Lemma 3 we showed that the worst case result of our heuristic algorithm is HEUR = n − 1 (which is the case in a path graph), hence \(HEUR \leq 3 OPT = n-1 \leq 3 \frac {n}{3}\).
□
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Toka, L. Ultra-Reliable and Low-Latency Computing in the Edge with Kubernetes. J Grid Computing 19, 31 (2021). https://doi.org/10.1007/s10723-021-09573-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10723-021-09573-z