Abstract
This paper discusses a novel approach to evaluate the real-time performance of computing platforms embedded in robotic systems. The motivation behind this work is the need for the robotic systems to meet their timing constraints, thus requiring time predictable real-time systems. We proposed a benchmark, named the Robotstone, which is an adaptation of a traditional real-time performance benchmark widely known in the real-time systems community. Thus, the presented work makes it easier for a robotic engineer to apply the benchmark in the modern robotics context by filtering out issues that do not matter on the robotics applications and adapting the benchmark’s relevant portions. The Robotstone has a set of experiments related to time-constrained application scenarios usually found in robotic systems. Each experiment defines an application-specific parameter that increases at every iteration until the system overloads. The real-time performance is then evaluated through the breakdown point, i.e., the system configuration in the application scenario when any functional or timing constraint is not met. The proposed toolset has been evaluated on two distinct platforms representing a class of embedded computing systems usually employed in robotic systems. Obtained results demonstrate the applicability of the Robotstone benchmark for a quick assessment of computing systems’ real-time performance often required on initial development stages of a robotic system.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Abella, J., Hernandez, C., Quiṅones, E., Cazorla, F.J., Conmy, P.R., Azkarate-Askasua, M., Perez, J., Mezzetti, E., Vardanega, T.: WCET analysis methods: Pitfalls and challenges on their trustworthiness. In: 10th IEEE International Symposium on Industrial Embedded Systems (SIES), pp 1–10. IEEE, Siegen (2015)
Amazon Web Services: The FreeRTOSTM Reference Manual. https://www.freertos.org/wp-content/uploads/2018/07/FreeRTOS Reference Manual_V10.0.0.pdf (2017). Accessed 23 July 2020
Ando, N., Suehiro, T., Kitagaki, K., Kotoku, T., Yoon, W.K.: RT-Middleware: Distributed component middleware for RT (Robot Technology). In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 3933–3938. IEEE, Edmonton (2005)
Anh, T.N.B., Tan, S.L.: Real-time operating systems for small microcontrollers. IEEE Micro 29(5), 30–45 (2009)
Arad, B., Kurtser, P., Barnea, E., Harel, B., Edan, Y., Ben-Shahar, O.: Controlled lighting and illumination-independent target detection for real-time cost-efficient applications. the case study of sweet pepper robotic harvesting. Sensors 19(6), 1390 (2019)
Asadi, K., Ramshankar, H., Pullagurla, H., Bhandare, A., Shanbhag, S., Mehta, P., Kundu, S., Han, K., Lobaton, E., Wu, T.: Vision-based integrated mobile robotic system for real-time applications in construction. Autom. Constr. 96, 470–482 (2018)
Axer, P., et al.: Building timing predictable embedded systems. ACM Trans. Embedded Comput. Syst. 13(4) (2014)
Barbalace, A., Luchetta, A., Manduchi, G., Moro, M., Soppelsa, A., Taliercio, C.: Performance comparison of VxWorks, Linux, RTAI, and Xenomai in a hard real-time application. IEEE Trans. Nucl. Sci. 55(1), 435–439 (2008)
Bardaro, G., Semprebon, A., Matteucci, M.: A use case in model-based robot development using Aadl and Ros. In: Proceedings of the International Workshop on Robotics Software Engineering, pp 9–16. ACM, New York (2018)
Barry, R.: Mastering the FreeRTOSTM Real Time Kernel – A Hands-On Tutorial Guide. https://www.freertos.org/wp-content/uploads/2018/07/161204%_Mastering_the_FreeRTOS_Real_Time_Kernel-A_Hands-On_Tutorial_Guide.pdf (2016). Accessed 23 July 2020
Bruyninckx, H.: Open robot control software: The OROCOS project. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation, vol. 3, pp 2523–2528. IEEE, Korea (2001)
Bruyninckx, H.: Open robot control software: The OROCOS project. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164), vol. 3, pp. 2523–2528 vol.3 (2001)
Bruyninckx, H.: OROCOS: Design and implementation of a robot control software framework. In: Proceedings of IEEE International Conference on Robotics and Automation. IEEE (2002)
Bucci, G., Carnevali, L., Ridi, L., Vicario, E.: Oris: A tool for modeling, verification and evaluation of real-time systems. Int J Softw Tools Technol Trans 12(5), 391–403 (2010)
Bäuml, B., Hirzinger, G.: When hard realtime matters: Software for complex mechatronic systems. Robot. Autonomous Syst. 56(1), 5–13 (2008). Human Technologies: ”Know-how”
Buschmann, F.: Pattern-oriented Software Architecture, a System of Patterns, vol. 1. Wiley, New Jersey (1996)
Butazzo, G.C.: Hard Real-Time Computing Systems: Predictable Scheduling Algorithms and Applications, 3rd edn. Springer, New York (2011)
Carpenter, J., Funk, S., Holman, P., Srinivasan, A., Anderson, J.H., Baruah, S.K.: A Categorization of real-time multiprocessor scheduling problems and algorithms. In: Handbook of Scheduling, pp 676–695. Chapman and Hall/CRC, Boca Raton (2004)
Curnow, H.J., Wichmann, B.A.: A synthetic benchmark. Comput. J. 19(1), 43–49 (1976)
Donohoe, P., Shapiro, R., Weiderman, N.: Hartstone Benchmark User’s Guide. Tech. Rep., Software Engineering Institute, Carnegie Mellon University Technical Report CMU-SEI- 90-TR-1 (1990)
Dudek, G., Jenkin, M.: Computational Principles of Mobile Robotics. Cambridge university press, Cambridge (2010)
Elkady, A., Sobh, T.: Robotics middleware: A comprehensive literature survey and attribute-based bibliography. J. Robot. 2012, 01–15 (2012)
Embedded: How to make C++ more real-time friendly. https://www.embedded.com/how-to-make-c-more-real-time-friendly/ (2014). Accessed 23 July 2020
Ferdinand, C., Heckmann, R., Langenbach, M., Martin, F., Schmidt, M., Theiling, H., Thesing, S., Wilhelm, R.: Reliable and Precise Wcet determination for a real-life processor. In: Henzinger, T.A., Kirsch, C.M. (eds.) Embedded Software, pp 469–485. Springer, Berlin (2001)
Flores, G., Zhou, S., Lozano, R., Castillo, P.: A vision and gps-based real-time trajectory planning for a mav in unknown and low-sunlight environments. J. Intell. Robot. Syst. 74(1-2), 59–67 (2014)
Gerum, P.: Xenomai - Implementing a RTOS emulation framework on GNU/Linux. https://design.ros2.org/articles/realtime_background.html (2004). Accessed 23 July 2020
Hammer, T., Bäuml, B.: The communication layer of the ardx software framework: Highly performant and realtime deterministic. J. Intell. Robot. Syst. 77(1), 171–185 (2015)
Heckmann, R., Langenbach, M., Thesing, S., Wilhelm, R.: The influence of processor architecture on the design and the results of wcet tools. Proc. IEEE 91(7), 1038–1054 (2003)
Jo, K., Kim, J., Kim, D., Jang, C., Sunwoo, M.: Development of autonomous car – Part II: A Case study on the implementation of an autonomous driving system based on distributed architecture. IEEE Trans. Ind. Electron. 62(8), 5119–5132 (2015)
Joseph, M., Pandya, P.: Finding response times in a Real-Time system. Comput. J. 29(5), 390–395 (1986)
Kamal, R.: Embedded Systems: Architecture, Programming and Design. Tata McGraw-Hill Education, New York (2011)
Kamenoff, N.I., Weiderman, N.H.: Hartstone distributed benchmark: Requirements and definitions. In: Real-Time Systems Symposium, 1991. Proceedings. Twelfth, pp 199–208. IEEE (1991)
Kerstens, R., Laurijssen, D., Steckel, J.: Ertis: A fully embedded real time 3D imaging sonar sensor for robotic applications. In: 2019 International Conference on Robotics and Automation (ICRA), pp 1438–1443. IEEE (2019)
Kiszka, J.: The real-time driver model and first applications. In: 7Th Real-Time Linux Workshop, Lille, France (2005)
Klein, M.H., Ralya, T., Pollak, B., Obenza, R., Harbour, M.G.: A Practioner’s Handbook for Real-Time Analysis: Guide to Rate Monotonic Analysis for Real-Time Systems. Kluwer Academic Publishers, Dordrecht (1993)
Kopetz, H.: Real-Time Systems: Design principles for distributed embedded applications. Springer Science & Business Media, Berlin (2011)
Krten, R.: Getting Started with QNX Neutrino 2: A Guide for Realtime Programmers. PARSE Software Devices, Kanata (1999)
Lee, E.A., Seshia, S.A.: Introduction to Embedded Systems - A Cyber-Physical Systems Approach, 2nd edn. MIT Press, Cambridge (2017)
Liu, C.L., Layland, J.W.: Scheduling algorithms for multiprogramming in a hard-real-time environment. J. ACM (JACM) 20(1), 46–61 (1973)
Liu, J.W.S.: Real-Time Systems. Prentice Hall, Upper Saddle River (2000)
Magyar, G., Sinčák, P., KrizsáN, Z.: Comparison study of robotic Middleware for robotic applications. In: Sinčák, P., Hartono, P., Virčíková, M., Vaščák, J., Jakša, R. (eds.) Emergent Trends in Robotics and Intelligent Systems, pp 121–128. Springer International Publishing, Cham (2015)
Maruyama, Y., Kato, S., Azumi, T.: Exploring the performance of ROS2. In: Proceedings of the 13th International Conference on Embedded Software, p 5. ACM (2016)
Metta, G., Fitzpatrick, P., Natale, L.: Yarp: Yet another robot platform. Int. J. Adv. Robot. Syst. 3(1), 8 (2006)
Muratore, L., Laurenzi, A., Hoffman, E.M., Rocchi, A., Caldwell, D.G., Tsagarakis, N.G.: Xbotcore: A real-time cross-robot software platform. In: 2017 First IEEE International Conference on Robotic Computing (IRC), pp 77–80. IEEE (2017)
Nise, N.S.: Control Systems Engineering, 6th edn. Wiley, New Jersey (2011)
Open Source Robotics Foundation: Introduction to real-time systems. https://design.ros2.org/articles/realtime_background.html (2019). Accessed 23 July 2020
Perathoner, S., Wandeler, E., Thiele, L., Hamann, A., Schliecker, S., Henia, R., Racu, R., Ernst, R., Harbour, M.G.: Influence of different system abstractions on the performance analysis of distributed real-time systems. In: Proceedings of the 7th ACM & IEEE International Conference on Embedded Software, pp 193–202 (2007)
Pinto, M.L., de Oliveira, A.S., Wehrmeister, M.A.: Real-time systems evaluation for robotics using the Hart-ROS benchmark. In: 2019 19th International Conference on Advanced Robotics (ICAR), pp 296–301. IEEE, Belo Horizonte (2019)
Quigley, M., Faust, J., Foote, T., Leibs, J.: Ros: An open-source robot operating system. In: International Conference on Robotics and Automation (ICRA) - Workshop on Open Source Software, vol. 3, p 5P (2009)
Rajkumar, R., Gagliardi, M., Sha, L.: The real-time Publisher/Subscriber Inter-Process communication model for distributed real-time systems: Design and implementation. In: 1st IEEE Real-Time Technology and Applications Symposium, pp 66–75. IEEE, United States (1995)
Redmon, J., Angelova, A.: Real-time grasp detection using convolutional neural networks. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp 131–1322. IEEE (2015)
RobMoSys EU H2020 Project (2017–2020). RobMoSys: Composable Models and Software for Robtics Systems - Towards an EU Digital Industrial Platform for Robotics. https://robmosys.eu (2020). Accessed 23 July 2020
Sha, L., Rajkumar, R., Lehoczky, J.P.: Priority inheritance protocols: an approach to real-time synchronization. IEEE Trans. Comput. 39(9), 1175–1185 (1990)
Siciliano, B., Khatib, O.: Springer Handbook of Robotics. Springer, Berlin (2016)
Siegwart, R., Nourbakhsh, I.R., Scaramuzza, D., Arkin, R.C.: Introduction to Autonomous Mobile Robots, 2nd edn. MIT press, Massachusetts (2011)
Stankovic, J.A.: Misconceptions about real-time computing: A serious problem for next-generation systems. Computer 21(10), 10–19 (1988)
Stankovic, J.A., Rajkumar, R.: Real-time operating systems. Real-Time Syst. 28(2-3), 237–253 (2004)
Subosits, J.K., Gerdes, J.C.: From the racetrack to the road: Real-time trajectory replanning for autonomous driving. IEEE Trans. Intell. Vehicles 4(2), 309–320 (2019)
Tanenbaum, A.S., Van Steen, M.: Distributed Systems: Principles and Paradigms. Prentice-Hall, New Jersey (2007)
Wang, M., Liu, J.N.K.: Fuzzy logic-based real-time robot navigation in unknown environment with dead ends. Robot. Auton. Syst. 56(7), 625–643 (2008)
Wehrmeister, M.A., de Freitas, E.P., Binotto, A.P.D., Pereira, C.E.: Combining aspects and object-orientation in model-driven engineering for distributed industrial mechatronics systems. Mechatronics 24(7), 844–865 (2014). https://doi.org/10.1016/j.mechatronics.201312.008
Weiderman, N.H., Kamenoff, N.I.: Hartstone uniprocessor benchmark: Definitions and experiments for real-time systems. Real-Time Syst. 4(4), 353–382 (1992)
Wilhelm, R., Engblom, J., Ermedahl, A., Holsti, N., Thesing, S., Whalley, D., Bernat, G., Ferdinand, C., Heckmann, R., Mitra, T., et al.: The worst-case execution-time problem – overview of methods and survey of tools. ACM Trans. Embedded Comput. Syst. (TECS) 7(3), 1–53 (2008)
Xu, Y., Fang, G., Chen, S., Zou, J.J., Ye, Z.: Real-time image processing for vision-based weld seam tracking in robotic gmaw. Int. J. Adv. Manufact. Technol. 73(9–12), 1413–1425 (2014)
Yaghmour, K., Masters, J., Ben-Yossef, G., Gerum, P.: Building Embedded Linux Systems, 2nd edn. O’Reilly, Sebastopol (2008)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Topical Collection on ICAR 2019 Special Issue
Rights and permissions
About this article
Cite this article
Pinto, M.L., Wehrmeister, M.A. & de Oliveira, A.S. Real-Time Performance Evaluation for Robotics. J Intell Robot Syst 101, 37 (2021). https://doi.org/10.1007/s10846-020-01301-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-020-01301-1