Abstract
Robot assistants need to interact with people in a natural way in order to be accepted into people’s day-to-day lives. We have been researching robot assistants with capabilities that include visually tracking humans in the environment, identifying the context in which humans carry out their activities, understanding spoken language (with a fixed vocabulary), participating in spoken dialogs to resolve ambiguities, and learning task procedures. In this paper, we describe a robot task learning algorithm in which the human explicitly and interactively instructs a series of steps to the robot through spoken language. The training algorithm fuses the robot’s perception of the human with the understood speech data, maps the spoken language to robotic actions, and follows the human to gather the action applicability state information. The robot represents the acquired task as a conditional procedure and engages the human in a spoken-language dialog to fill in information that the human may have omitted.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Arkin R (1987) Motor schema based navigation for a mobile robot: an approach to programming by behavior. In: Proceedings of the 1987 IEEE international conference on robotics and automation, pp 264–271
Bentivegna D, Atkeson C, Cheng G (2003) Learning from observation and practice at the action generation level. In: IEEE international conference on humanoid robots, Karlsruhe and Munich
Biermann AW, Guinn CI, Smith DRHRW (1994) Efficient collaborative discourse: a theory and its implementation. In: Proc. ARPA human language technology workshop ’93, Princeton, NJ, pp 177–182
Breazeal C, Hoffman G, Lockerd A (2004) Teaching and working with robots as a collaboration. In: The third international conference on autonomous agents and multi-agent systems AAMAS 2004, New York, NY, pp 1028–1035
Bruce J, Veloso M (2003) Fast and accurate vision-based pattern detection and identification. In: Proceedings of the 2003 IEEE international conference on robotics and automation, Taiwan
Burgard W, Cremers A, Fox D, Hähnel D, Lakemeyer G, Schulz D, Steiner W, Thrun S (1999) Experiences with an interactive museum tour-guide robot. Artifi Intell 114(1–2): 3–55
Clarke HH (1996) Using language. Cambridge University Press, New York
Ehrenmass M, Zöllner R, Rogalla O, Vacek S, Dillmann R (2003) Observation in programming by demonstration: training and execution environment. In: IEEE international conference on humanoid robots, Karlsruhe and Munich
Feil-Seifer DJ, Matarić MJ (2005) Defining socially assistive robotics. In: International conference on rehabilitation robotics, Chicago, pp 465–468
Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robot Auton Systems 42: 143–166
Guinn CI (1996) Mechanisms for mixed-initiative human-computer collaborative discourse. In: Joshi A, Palmer M(eds) Proceedings of the thirty-fourth annual meeting of the association for computational linguistics. Morgan Kaufmann, San Francisco, pp 278–285
Guinn CI (1998) An analysis of initiative selection in collaborative task-oriented discourse. User Model User-Adapted Inter 8(3–4): 255–314
Lenser S, Bruce J, Veloso M (2002) A modular hierarchical behavior-based architecture. In: Birk A, Coradeschi S, Tadokoro S(eds) RoboCup-2001: the fifth RoboCup competitions and conferences. Springer, Berlin
Lesh N, Rich C, Sidner C (1999) Using plan recognition in human-computer collaboration. In: Proceedings of the seventh international conference on user modeling
Martignoni AJ, Smart WD (2004) Programming robots using high-level task descriptions. In: Supervisory control of learning and adaptive systems: papers from the 2004 AAAI workshop, pp 49–54
Nicolescu M, Matarić M (2001) Experience-based representation construction: learning from human and robot teachers. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, Maui, pp 740–745
Nicolescu M, Matarić M (2003) Natural methods for robot task learning: Instructive demonstration, generalization and practice. In: Proceedings of the second international joint conference on autonomous agents and multi-agent systems, Melbourne
Oblinger D, Castelli V, Bergman L (2006) Augmentation-based learning: combining observations and user edits for programming by demonstration. In: Proceedings of the international conference on intelligent user interfaces, pp 202–209
Perzanowski D, Schultz A, Adams W, Marsh E, Bugajska M (2001) Building a multimodal human-robot interface. IEEE Intell Systems 16(1): 16–21
Pineau J, Montemerlo M, Pollack M, Roy N, Thrun S (2003) Towards robotic assistants in nursing homes: challenges and results. Robot Auton Systems 42(31): 271–281
Rich C, Sidner CL (1998) COLLAGEN: a collaboration manager for software interface agents. User Model User-Adapted Inter 8(3–4): 315–350
Rybski PE, de la Torre F, Patil R, Vallespi C, Veloso MM, Browning B (2004) Cameo: the camera assisted meeting event observer. In: Proceedings of the 2004 IEEE international conference on robotics and automation, New Orleans
Saunders J, Nehaniv CL, Dautenhahn K (2006) Teaching robots by moulding behavior and scaffolding the environment. In: Human–robot interaction, Salt Lake City
Simmons R, Goldberg D, Goode A, Montemerlo M, Roy N, Sellner B, Urmson C, Schultz A, Abramson M, Adams W, Atrash A, Bugajska M, Coblenz M, MacMahon M, Perzanowski D, Horswill I, Zubek R, Kortenkamp D, Wolfe B, Milam T, Maxwell B (2003) Grace: an autonomous robot for the aaai robot challenge. AI Magaz 42(2): 51–72
Skubic M, Perzanowski D, Blisard S, Schultz A, Adams W, Bugajska M, Brock D (2004) Spatial language for human-robot dialogs. IEEE Trans Systems, Man Cybern Part C 34(2): 154–167
Sofge D, Trafton JG, Cassimatis N, Perzanowski D, Bugajska M, Adams W, Schultz AC (2004) Human–robot collaboration and cognition with an autonomous mobile robot. In: Groen F, Amato N, Bonarini A, Yoshida E, Kröse B (eds) In: Proceedings of the 8th conference on intelligent autonomous systems (IAS-8). IOS Press, pp 80–87
Willeke T, Kunz C, Nourbakhsh I (2001) The history of the mobot museum robot series: An evolutionary study. In: Proceedings of FLAIRS 2001
Winner E, Veloso M (2002) Analyzing plans with conditional effects. In: Proceedings of the sixth international conference on artificial intelligence planning systems, Toulouse
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Rybski, P.E., Stolarz, J., Yoon, K. et al. Using dialog and human observations to dictate tasks to a learning robot assistant. Intel Serv Robotics 1, 159–167 (2008). https://doi.org/10.1007/s11370-008-0016-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11370-008-0016-5