Skip to main content
SpringerLink
Log in

Shared-Control Teleoperation Paradigms on a Soft-Growing Robot Manipulator

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Semi-autonomous telerobotic systems allow both humans and robots to exploit their strengths while enabling personalized execution of a remote task. For soft robots with kinematic structures dissimilar to those of human operators, it is unknown how the allocation of control between the human and the robot changes the performance. This work presents a set of interaction paradigms between a human and a remote soft-growing robot manipulator, with demonstrations in both real and simulated scenarios. The soft robot can grow and retract by eversion and inversion of its tubular body, a property we exploit in the interaction paradigms. We implemented and tested six different human-robot interaction paradigms, with full teleoperation at one extreme and gradually adding autonomy to various aspects of the task execution. All paradigms are demonstrated by two experts and two naive operators. Results show that humans and the soft robot manipulator can effectively split their control along different degrees of freedom while acting simultaneously to accomplish a task. In the simple pick-and-place task studied in this work, performance improves as the control is gradually given to the robot’s autonomy, especially when the robot can correct certain human errors. However, human engagement is maximized when the control over a task is at least partially shared. Finally, when the human operator is assisted by haptic guidance, which is computed based on soft robot tip position errors, we observed that the improvement in performance is dependent on the expertise of the human operator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Code or data availability

The simulator scene and its scripts, as well as the C++ software implementing the interaction modules described in the text, are publicly released at https://github.com/mrslvg/sgm-sim/.

References

  1. de Graaf, M.M., Allouch, S.B., van Dijk, J. (2016) Long-term acceptance of social robots in domestic environments: Insights from a user’s perspective. In: AAAI Spring Symposia

  2. Scalise, R., Bisk, Y., Forbes, M., Yi, D., Choi, Y., Srinivasa, S. (2018) Balancing shared autonomy with human-robot communication. arXiv:1805.07719

  3. Schilling, M., Kopp, S., Wachsmuth, S., Wrede, B., Ritter, H., Brox, T., Nebel, B., Burgard, W. (2016) Towards a multidimensional perspective on shared autonomy. In: AAAI fall symposium series, pp. 338–344

  4. Coad, M.M., Thomasson, R.P., Blumenschein, L.H., Usevitch, N.S., Hawkes, E.W., Okamura, A.M.: Retraction of soft growing robots without buckling. IEEE Robotics & Automation Letters 5(2), 2115–2122 (2020)

    Article  Google Scholar 

  5. El-Hussieny, H., Mehmood, U., Mehdi, Z., Jeong, S.-G., Usman, M., Hawkes, E.W., Okamura, A.M., Ryu, J.-H. (2018) Development and evaluation of an intuitive flexible interface for teleoperating soft growing robots. In: IEEE/RSJ international conference on intelligent robots and systems, pp. 4995–5002

  6. Mörtl, A., Lawitzky, M., Kucukyilmaz, A., Sezgin, M., Basdogan, C., Hirche, S.: The role of roles: Physical cooperation between humans and robots. The International Journal of Robotics Research 31(13), 1656–1674 (2012)

    Article  Google Scholar 

  7. Corrales, J., Gomez, G.G., Torres, F., Perdereau, V.: Cooperative tasks between humans and robots in industrial environments. International Journal of Advanced Robotic Systems 9(3), 94 (2012)

    Article  Google Scholar 

  8. Pitzer, B., Styer, M., Bersch, C., DuHadway, C., Becker, J. (2011) Towards perceptual shared autonomy for robotic mobile manipulation. In: IEEE international conference on robotics and automation, pp. 6245–6251

  9. Kortenkamp, D., Keirn-Schreckenghost, D., Bonasso, R.P.: Adjustable control autonomy for manned space flight. IEEE Aerospace Conference (AeroConf) 7, 629–640 (2000)

    Google Scholar 

  10. Sellner, B., Simmons, R., Singh, S. (2005) User modelling for principled sliding autonomy in human-robot teams. In: Multi-Robot Systems. From Swarms to Intelligent Automata vol. 3, pp. 197–208. Springer

  11. Dias, M.B., Kannan, B., Browning, B., Jones, E., Argall, B., Dias, M.F., Zinck, M., Veloso, M., Stentz, A. (2008) Sliding autonomy for peer-to-peer human-robot teams. In: International Conference on Intelligent Autonomous Systems, pp. 332–341

  12. Rieffel, J., Knox, D., Smith, S., Trimmer, B.: Growing and evolving soft robots. Artificial life 20(1), 143–162 (2014)

    Article  Google Scholar 

  13. Coad, M.M., Blumenschein, L.H., Cutler, S., Zepeda, J.A.R., Naclerio, N.D., El-Hussieny, H., Mehmood, U., Ryu, J.-H., Hawkes, E.W., Okamura, A.M.: Vine robots: Design, teleoperation, and deployment for navigation and exploration. IEEE Robotics & Automation Magazine 27(3), 120–132 (2020)

    Article  Google Scholar 

  14. Hawkes, E.W., Blumenschein, L.H., Greer, J.D., Okamura, A.M.: A soft robot that navigates its environment through growth. Science Robotics 2(8), 3028 (2017)

    Article  Google Scholar 

  15. Sadeghi, A., Mondini, A., Mazzolai, B.: Toward self-growing soft robots inspired by plant roots and based on additive manufacturing technologies. Soft Robotics 4(3), 211–223 (2017)

    Article  Google Scholar 

  16. Tsukagoshi, H., Arai, N., Kiryu, I., Kitagawa, A.: Tip growing actuator with the hose-like structure aiming for inspection on narrow terrain. International Journal of Automation Technology 5(4), 516–522 (2011)

    Article  Google Scholar 

  17. Dehghani, H., Welch, C.R., Pourghodrat, A., Nelson, C.A., Oleynikov, D., Dasgupta, P., Terry, B.S.: Design and preliminary evaluation of a self-steering, pneumatically driven colonoscopy robot. Journal of Medical Engineering & Technology 41(3), 223–236 (2017)

    Article  Google Scholar 

  18. Blumenschein, L.H., Gan, L.T., Fan, J.A., Okamura, A.M., Hawkes, E.W.: A tip-extending soft robot enables reconfigurable and deployable antennas. IEEE Robotics & Automation Letters 3(2), 949–956 (2018)

    Article  Google Scholar 

  19. Luong, J., Glick, P., Ong, A., deVries, M.S., Sandin, S., Hawkes, E.W., Tolley, M.T. (2019) Eversion and retraction of a soft robot towards the exploration of coral reefs. In: IEEE International Conference on Soft Robotics (RoboSoft), pp. 801–807

  20. Naclerio, N.D., Hubicki, C.M., Aydin, Y.O., Goldman, D.I., Hawkes, E.W. (2018) Soft robotic burrowing device with tip-extension and granular fluidization. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 5918–5923. IEEE

  21. Stroppa, F., Luo, M., Yoshida, K., Coad, M.M., Blumenschein, L.H., Okamura, A.M. (2020) Human interface for teleoperated object manipulation with a soft growing robot. In: IEEE International Conference on Robotics and Automation, pp. 726–732

  22. Goodrich, M.A., Schultz, A.C.: Human-robot interaction: a survey. Foundations and Trends in Human-Computer Interaction 1(3), 203–275 (2008)

    Article  MATH  Google Scholar 

  23. Selvaggio, M., Cognetti, M., Nikolaidis, S., Ivaldi, S., Siciliano, B.: Autonomy in physical human-robot interaction: A brief survey. IEEE Robotics and Automation Letters 6(4), 7989–7996 (2021)

    Article  Google Scholar 

  24. Yang, G.-Z., Cambias, J., Cleary, K., Daimler, E., Drake, J., Dupont, P.E., Hata, N., Kazanzides, P., Martel, S., Patel, R.V., Santos, V.J., Taylor, R.H.: Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy. Science Robotics 2(4), 8638 (2017)

    Article  Google Scholar 

  25. Kanda, T., Ishiguro, H.: Human-robot Interaction in Social Robotics. CRC Press (2017)

    Book  Google Scholar 

  26. Schilling, M., Burgard, W., Muelling, K., Wrede, B., Ritter, H.: Shared autonomy—learning of joint action and human-robot collaboration. Frontiers in Neurorobotics 13, 16 (2019)

  27. Bruemmer, D.J., Dudenhoeffer, D.D., Marble, J.L.(2002) Dynamic-autonomy for urban search and rescue. In: AAAI mobile robot competition, pp. 33–37

  28. Desai, M., Yanco, H.A. (2005) Blending human and robot inputs for sliding scale autonomy. In: IEEE International Workshop on Robot and Human Interactive Communication (ROMAN), pp. 537–542

  29. Anderson, S., Peters, S., Iagnemma, K., Overholt, J. (2010) Semi-autonomous stability control and hazard avoidance for manned and unmanned ground vehicles. In: The 27th Army Science Conference, pp. 1–8

  30. Selvaggio, M., Giordano, P.R., Ficuciello, F., Siciliano, B. (2019) Passive task-prioritized shared-control teleoperation with haptic guidance. In: IEEE International Conference on Robotics and Automation, pp. 430–436

  31. Kofman, J., Wu, X., Luu, T.J., Verma, S.: Teleoperation of a robot manipulator using a vision-based human-robot interface. IEEE Transactions on Industrial Electronics 52(5), 1206–1219 (2005)

    Article  Google Scholar 

  32. Griffin, W.B., Provancher, W.R., Cutkosky, M.R. (2005) Feedback strategies for telemanipulation with shared control of object handling forces. Presence: Teleoperators & Virtual Environments 14(6), 720–731

  33. Li, M., Okamura, A.M. (2003)Recognition of operator motions for real-time assistance using virtual fixtures. In: IEEE Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 125–131

  34. Kragic, D., Marayong, P., Li, M., Okamura, A.M., Hager, G.D.: Human-machine collaborative systems for microsurgical applications. The International Journal of Robotics Research 24(9), 731–741 (2005)

    Article  Google Scholar 

  35. Shen, J., Ibanez-Guzman, J., Ng, T.C., Chew, B.S.: A collaborative-shared control system with safe obstacle avoidance capability. IEEE Conference on Robotics, Automation and Mechatronics 1, 119–123 (2004)

    Google Scholar 

  36. Dragan, A.D., Srinivasa, S.S.: A policy-blending formalism for shared control. The International Journal of Robotics Research 32(7), 790–805 (2013)

    Article  Google Scholar 

  37. Javdani, S., Srinivasa, S.S., Bagnell, J.A. (2015) Shared autonomy via hindsight optimization. In: Robotics Science and Systems. NIH Public Access

  38. Crandall, J.W., Goodrich, M.A. (2002) Characterizing efficiency of human robot interaction: A case study of shared-control teleoperation. In: IEEE/RSJ international conference on intelligent robots and systems, pp. 1290–1295

  39. Aarno, D., Ekvall, S., Kragic, D. (2005) Adaptive virtual fixtures for machine-assisted teleoperation tasks. In: IEEE International Conference on Robotics and Automation, pp. 1139–1144

  40. Selvaggio, M., Notomista, G., Chen, F., Gao, B., Trapani, F., Caldwell, D. (2016) Enhancing bilateral teleoperation using camera-based online virtual fixtures generation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1483–1488

  41. Selvaggio, M., Abi-Farraj, F., Pacchierotti, C., Robuffo Giordano, P., Siciliano, B.: Haptic-based shared-control methods for a dual-arm system. IEEE Robotics & Automation Letters 3(4), 4249–4256 (2018)

    Article  Google Scholar 

  42. Selvaggio, M., Ghalamzan E., A.M., Moccia, R., Ficuciello, F., Siciliano, B. (2019) Haptic-guided shared control for needle grasping optimization in minimally invasive robotic surgery. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3617–3623

  43. O’Malley, M.K., Gupta, A., Gen, M., Li, Y.(2006) Shared control in haptic systems for performance enhancement and training. Journal of Dynamic Systems, Measurement, and Control 128

  44. Squeri, V., Basteris, A., Sanguineti, V. (2011) Adaptive regulation of assistance ‘as needed’ in robot-assisted motor skill learning and neuro-rehabilitation. In: IEEE International Conference on Rehabilitation Robotics, pp. 1–6

  45. Guidali, M., Schlink, P., Duschau-Wicke, A., Riener, R. (2011) Online learning and adaptation of patient support during adl training. In: IEEE International Conference on Rehabilitation Robotics, pp. 1–6

  46. Stroppa, F., Marcheschi, S., Mastronicola, N., Loconsole, C., Frisoli, A. (2017) Online adaptive assistance control in robot-based neurorehabilitation therapy. In: IEEE International Conference on Rehabilitation Robotics, pp. 628–633

  47. Greer, J.D., Morimoto, T.K., Okamura, A.M., Hawkes, E.: A soft, steerable continuum robot that grows via tip extension. Soft Robotics 6(1), 95–108 (2019)

  48. Coad, M.M. (2021) Design, modeling, and control of vine robots for exploration of unknown environments. PhD thesis, Stanford University

  49. Gruebele, A.M., Zerbe, A.C., Coad, M.M., Okamura, A.M., Cutkosky, M.R. (2020) Distributed sensor networks deployed using soft growing robots. In: IEEE International Conference on Soft Robotics (RoboSoft), p

  50. Rohmer, E., Singh, S.P., Freese, M. (2013) V-rep: A versatile and scalable robot simulation framework. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1321–1326

  51. Nuño, E., Basañez, L., Ortega, R.: Passivity-based control for bilateral teleoperation: A tutorial. Automatica 47(3), 485–495 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  52. Chen, J.Y., Haas, E.C., Barnes, M.J. (2007) Human performance issues and user interface design for teleoperated robots. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37(6), 1231–1245

Download references

Funding

This work was supported in part by Toyota Research Institute (TRI) and National Science Foundation grant 2024247. TRI provided funds to assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.

Author information

Authors and Affiliations

  1. Computer Engineering Department, Kadir Has University, 34083, İstanbul, Turkey

    Fabio Stroppa

  2. Electrical Engineering and Information Technology Department, University of Naples Federico II, 80125, Naples, Italy

    Mario Selvaggio

  3. Mechanical Engineering Department, Stanford University, 94305, Stanford, CA, USA

    Nathaniel Agharese & Allison M. Okamura

  4. School of Mechanical and Materials Engineering, Washington State University, 99163, Pullman, WA, USA

    Ming Luo

  5. School of Mechanical Engineering, Purdue University, 47907, West Lafayette, IN, USA

    Laura H. Blumenschein

  6. Mechanical Engineering Department, UC Santa Barbara, 93105, Santa Barbara, CA, USA

    Elliot W. Hawkes

Authors
  1. Fabio Stroppa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mario Selvaggio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nathaniel Agharese
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Laura H. Blumenschein
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Elliot W. Hawkes
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Allison M. Okamura
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Fabio Stroppa designed and developed the interaction paradigms, designed and developed the overall system (motion tracking, computer vision, network communication, interfaces), designed and developed the experimental setup, designed and built the circuit board for the soft robot, assembled the soft robot, designed and implemented the inverse kinematics for the soft robot, supervised the real-scenario demonstrations, analyzed the data, and wrote the manuscript. Mario Selvaggio designed and implemented the virtual model of the soft robot for simulation, ran the statistical analysis for the parameter tuning for haptic rendering, supervised the simulated-scenario demonstrations, and partially wrote the manuscript. Nathaniel Agharese and Laura H. Blumenschein contributed to implementing the closed-loop control of the system and reviewed the manuscript. Ming Luo built the prototype of the soft robot and reviewed the manuscript. Elliot W. Hawkes reviewed the manuscript. Allison M. Okamura supervised the project and reviewed the manuscript.

Corresponding author

Correspondence to Fabio Stroppa.

Ethics declarations

Ethics approval

This is an observational study. The Institutional Review Board of Stanford University has confirmed that no ethical approval is required.

Consent to participate

Informed consent was obtained from all individual participants included in the study

Consent for publication

No individual person’s data in any form is disclosed in the paper.

Conflicts of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 17833 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stroppa, F., Selvaggio, M., Agharese, N. et al. Shared-Control Teleoperation Paradigms on a Soft-Growing Robot Manipulator. J Intell Robot Syst 109, 30 (2023). https://doi.org/10.1007/s10846-023-01919-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-023-01919-x

Keywords

Search

Navigation