Abstract
To realize the autonomous exploration and the cooperation behaviors of robots in the unknown environment, an improved internal model to evoke robots actions using a psychological theory of Russell was proposed in our previous work. The improved model is based on an affect-action model proposed by Ide and Nozawa group whose basic principle is to control the movement of robots by the degrees of “pleasure” and “arousal” of one’s own and the observation of others. To overcome the phenomena of “deadlock” and adapt to the complicated environment, “curiosity” factor is introduced into the basic model, and the action function is improved to be dynamically. This paper provides experimental comparison between the conventional model and our improved model with goal-exploration simulations. The results showed that only robots with the improved model moved dynamically and successfully reached at multiple goal areas avoiding local traps and obstacles in the complicated environment.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
Autonomous robot is defined as a robot that is able to act adaptively to the unknown or dynamic environments. To acquire the autonomic behaviors, an internal model of robot needs to be designed not only with the ability of information processing in response to external stimuli, but also with the ability of self-control. In another word, adaptive behaviors driven by the internal model depend on the observation of both external environment and internal state. Behaviors or actions of autonomous robots result in the change of its environment; meanwhile, the variable states of the environment keep affecting the internal decision system of robots. So how to design an intelligent and powerful “machine conscious system” to yield constant adaptive behaviors of an autonomous system or robot is considered more and more recently [1].
When multiple autonomous robots (agents) exist in the unknown or dynamic environment, “collective behaviors” or “cooperative behaviors” are expected to realize their common goals. In [2], a survey of multiple-robot system studies is given, and it is advocated that “group architecture”, “resource conflict”, “learning”, and “the origin of cooperation” using biological analogies and fundamental theories of social science are serviceable for the design of autonomous robots. In [3], cooperative behavior acquisition of mobile robots is realized with reinforcement learning which is inspired by behaviorist psychology. We also proposed a neuro-fuzzy reinforcement learning system for swarm formation and adaptive swarm behavior acquisition of the swarm robots in the previous work [4–7]. Meanwhile, Ide and Nozawa group proposed an internal model in [8, 9], which drives autonomous robots avoiding obstacles and exploring a goal in the unknown environments using a psychological model proposed by Russell in [10, 11]. Russell’s “circumplex model of affect” assigns 8 kinds of major emotions on a two-dimensional map, and by psychological analysis of evidences, affective states such as pleasure, excitement, arousal, distress, misery, depression, sleepiness, and contentment are categorized orderly in a pleasure-arousal space (Fig. 1). The emotion model of robots given by [8] uses the factors of “pleasure” and “arousal”, those are orthogonal axes concerning with all affective states, to decide a series of behavior rules for the autonomous robots. Degrees (values) of “pleasure” and “arousal” control the velocity of robots and obstacles. And these states of other robots are also observed and affect the values of “pleasure” or “sleepiness” of the observer (robot). In [9], the emotion model is applied to cooperative work acquisition of autonomous robots. However, when we executed the simulation of goal-exploration problem using the model, some practical problems were found such as dead-lock, limitation of vision depth, and restriction of speed. So we proposed an improved internal model in [12] with a new factor: “curiosity”, which is not included in Russell’s mental map.
In this paper, we intend to compare and analyze Ide and Nozawa’s model and our improved model by goal-exploration problem simulation and evaluation more in detail. Two kinds of unknown environments for two affection-driven robots are designed: a simple environment with one goal to explore and a complex environment with three goals. Different delay times of the start of robots are also set to observe the change of behaviors and internal states of robots. The rest of the paper is organized as follows. In section “An Improved Internal Model of Autonomous Robot”, conventional model and improved model are described. Comparison of goal-exploration simulations with the two models in the different environment is presented in section “Simulation Experiments”. Finally, the conclusion is given in section “Conclusion”.
An Improved Internal Model of Autonomous Robot
Russell’s emotion model [10] is shown in Fig. 1. The main difference from traditional psychological analyses of affect is that only pleasure and arousal dimensions are stressed in the model; meanwhile, conventionally a set of dimensions such as displeasure, distress, depression, excitement, and so on were considered independently. According to 28 stimulus words presented to 36 young peoples, [10] described the emotion categories in the circular ordering. In [11], it is pointed that the circumplex model suggested a clear structure and has large heuristic value for the evaluation of emotion. The group of Ide and Nozawa used the concept of the circumplex model to design an emotion model to evoke interactions or cooperative behaviors of multiple autonomous robots [8, 9]. Furthermore, Oudeyer and Kaplan used a curiosity model that considered the influence of time factor to raise the motivation of adaptive activities of robots [13]. In this Section, the emotion model of robots is introduced at first and an improved internal model including the emotional concept and a novel calculation method of curiosity is proposed.
A Conventional Emotion Model for Robots
In a goal-exploration problem, robots move to search the goal and avoiding to obstacles and other robots in the unknown 2-D environments. In the conventional emotion model [8], information of local environment around the robot is obtained by the observation and the information determines the degree of emotional vectors: “pleasure” and “arousal”, which cause the motion of robot.
First of all, the position vector \( \user2{R}_{\user2{i}} (\user2{t}) \) of robot i at time t is given by:
where t: step (time); \( \user2{V}{}_{\user2{i}}(\user2{t}) \): velocity vector of robot i at time t .
The velocity \( \user2{V}{}_{\user2{i}}(\user2{t}) \) of robot i is decided by the degree of “pleasure” of itself \( Pv{}_{ij} \)and other’s \( Pv{}_{ji} \):
where \( l_{1} ,l_{2} \): emotional influence parameters; \( Pv{}_{ji} \): influence from robot j to robot i; \( Pv{}_{ij} \): influence from robot i to robot j.
The negative coefficient of \( Pv{}_{ji} \) results in avoiding the crowd of robots and supports robot i’s behavior of exploration.
The influences between the robots are in direct ratios to the distances (Euclidean) between them, calculated by:
respectively, where \( r_{ij} ,r_{ji} \) are distances (Euclidean) between robot i and j. The degree of “pleasure” of each robot is decided by the depth of the vision:
\( R_{pv} \): rate of the change of “pleasure” \( (0 \le R_{pv} \le 1) \); \( d_{o} \): distance (scalar) from robot i to the nearest obstacle; \( d_{r} \): distance (scalar) from robot i to the nearest robot; \( D \): threshold value (scalar) of depth of the vision; \( e_{p} \): positive coefficient.
The threshold value of vision depth decides the change of “pleasure” and is decided by the degree of “arousal” \( Av \):
where \( Av \): degree of “arousal”; \( K \): bias of the vision depth; \( \alpha \): positive coefficients.
The degree of “arousal” \( Av \) is also influenced by the situation of the existence of other robots beyond its vision:
where \( \, e_{a} : \) coefficients; \( R_{Av} \): rate of the change of “arousal” \( (0 \le R_{Av} \le 1 ) \).
According to definitions described above, robots move in the environment as follows:
-
(1)
Local information is obtained within the environment of vision;
-
(2)
The degree of “arousal” is influenced by the distance to other robots (Eqs. 8, 9);
-
(3)
A robot comes up to other robots that are “pleasure” appearing within the vision and comes off to ones in the opposite case (Eq. 2);
-
(4)
A robot comes up to other robots, when it is in the state of “pleasure”, and comes off to the others in the opposite case (Eq. 2);
-
(5)
The degree of “pleasure” is reduced, when obstacles or other robots are observed, and increased in the opposite case (Eqs. 5, 6);
-
(6)
The degree of “arousal” is increased, when other robots are observed, and reduced in the opposite case (Eqs. 8, 9).
According to these rules, patterns of behaviors of robots appear as follows:
-
(1)
Robots in “pleasure” state attract each other and come up to each other closely (Fig. 2a);
-
(2)
A robot in “pleasure” state moves to a direction to toward the goal and causes others to follow it (Fig. 2b);
-
(3)
Robots in “displeasure” state go away from each other (Fig. 2c).
An Improved Internal Model for Autonomous Robots
Using the conventional model described in the section “A Conventional Emotion Model for Robots”, we performed simulation experiments and observed kinds of results such as robots successfully attracted each other, avoided to obstacles and achieved at the goal(s) of exploration, or failed to attract each other, failed to achieve on the multiple goal areas in more complicated environment.
The reasons of the failed cases may be considered as follows:
-
(1)
Bias of the vision was set inadequately. Too large value of K caused the internal state of robot dropped into “displeasure” easily.
-
(2)
The time that robots influent each other was too short because of too high velocity.
-
(3)
There was a trend that the degree of pleasure reduced more easily than increased.
-
(4)
Low degree of pleasure of all robots caused low degree of arousal of robots and the case resulted in all robots dropped into the state of sleepiness, the behaviors of exploration disappeared.
To overcome these problems and to raise the motivation of exploration, in [12], we proposed to add new rules into the emotion model and adopt a new mental factor “curiosity” into the calculation of the velocity vector as following:
-
(1)
Limit bounds of the depth of vision: K min < K < K max;
-
(2)
Limit a maximum value of velocity: V i (t) < V max;
-
(3)
Make the change of emotion factor “pleasure” to be dynamical, i.e., using Eqs. 10 and 11 instead of Eq. 5.
where \( \mu , \, M , { }N, \, \beta \) are positive parameters.
-
(4)
“Curiosity” means two situations concerning with the change of the internal state of robots:
-
(i)
Robot i keeps to search the goals k (k = 1, 2,…,k,…,K) before it arrives at them and after it arrives at one goal k, then its “curiosity” to the goal is reduced eventually;
-
(ii)
During robot i exploring the environment, when it crushes to obstacles, its “curiosity” is reduced eventually.
-
(i)
Equation 12 defines the “curiosity”, and Eq. 13 builds an improved internal model of autonomous robots:
where \( Cv(t) \) is the degree of “curiosity” in the improved internal model, \( I{}_{k} \) is the positive parameter for different goal k and coefficients \( \lambda_{1} , \, \lambda_{2} \), \( l_{3} > 0 \).
Simulation Experiments
To exam the improved internal model proposed in section “An Improved Internal Model for Autonomous Robots”, computer simulation experiments were performed using two kinds of environments for multiple robots goal exploration. Goal position is supposed to be known by the robots; however, the route to the goal (or goals) is unknown. Robots are required to move to goal area(s) avoiding crush to the obstacle or each other. The movements of robots are the behaviors driven by the internal models using affective factors.
In section “Single Goal in Simple Environment”, a simple environment with single goal was used for two autonomous robots; meanwhile, in section “Multiple Goals in Complicated Environment”, a complicated environment (maze-like) with multiple goals was used. The number of robots was increased to 5 in simulations; conveniently, here, we show the results of two robots for analyzing the effectiveness of the theory.
Single Goal in Simple Environment
The size of a two-dimensional exploring space is 500 × 500, an obstacle exists in the center of the square, and two robots start from the corner of left-down to search a goal area located at the corner of right-up: the environment of simulation is shown in Fig. 3. To compare with simulations reported in [8], two cases of timing of start of robots were also executed in our experiments:
-
(i)
Two robots started at the same time;
-
(ii)
One started after another one started 200 steps. All parameters were set as shown in Table 1.
-
(i)
Simulation results of two robots started at the same time are shown in Fig. 4. The total exploration time using the improved model was 134 steps (Fig. 4b), which is less than the time of conventional model 151 steps (Fig. 4a). The change of pleasure degree of the robots showed similar as shown as in Fig. 5.
-
(ii)
Simulation results of two robots started at the same time are shown in Fig. 6. The total exploration time using the improved model was 335 steps (Fig. 6b), which is more than the time of conventional model 291 steps (Fig. 6b). The change of pleasure degree of the robots showed similar as shown as in Fig. 7.
Multiple Goals in Complicated Environment
The size of a two-dimensional exploring space is 500 × 500, multiple obstacles exist in the different positions of the square, two robots start from two different positions to search three goal areas located at the different positions: the environment of simulation is shown in Fig. 8. Two cases of timing of start of robots were also executed in this experiment:
-
(i)
Two robots started at the same time;
-
(ii)
One started after another one started 200 steps.
All parameters were set as shown in Table 2, and the limitation of steps of a trial was set to 2,000 steps.
-
(i)
Simulation results of two robots started at the same time are shown in Fig. 9a, b. Robots with conventional model stopped exploration at step 15, for the kinds of reasons such as obstacles and multiple goals (Fig. 9a); meanwhile, in the occasion of the improved model showed active exploration and reached at all three goals within 2,000 steps limitation of time (Fig. 9b).
-
(ii)
Simulation results of two robots started at the different time are shown in Fig. 10a, b. Robots with conventional model also stopped exploration at step 214, failed to arrive at any goal (Fig. 10a). Robots with the improved model also showed active exploration; however, one failed to reach at Goal 3 in the limitation of 2,000 steps (Fig. 10b).
The change of the degree of pleasure, as curves depicted in Figs. 11 and 12 respective to Figs. 9 and 10, showed the difference of the internal state between conventional model and improved model. More dynamical activity was observed in the case of our improved model.
Discussions
According to the simulation results reported in sections “Single Goal in Simple Environment” and “Multiple Goals in Complicated Environment”, the improved model showed its effectiveness in different environments and its superiority to the conventional model in the situation of exploring multiple goals in the complicated environment. A comparison of these results is given by Table 3.
Conclusion
To deal with dynamical complex environment and acquire collaborative behaviors of autonomous robots, “mental” states of robots play important roles during the decision process of actions. An emotion-curiosity-driven behavior model is proposed for the cooperative exploration activity of multiple robots. Machine consciousness including pleasure, arousal, and curiosity motivates robots to control the velocities of exploration movement and keep the activeness of exploration in the unknown environments. Kinds of simulations were executed, and the results showed the effectiveness of the proposed model. Other intelligent factors such as memory function and learning ability which often adopted in systems of biological analogy are expected to be composed with the internal model to raise the exploration efficiency in the future.
References
Aleksander I. Designing conscious systems. Cognit Comput. 2009;1:22–8.
Cao YU, Fukunaga AS, Kahng AB. Cooperative mobile robotics: antecedents and directions. Auton Robots. 1997;4:7–27.
Asada M, Uchibe E, Hosoda K. Cooperative behavior acquisition for mobile robots in dynamically changing real worlds via vision-based reinforcement learning and development. Artif Intell. 1999;110:275–92.
Kuremoto T, Obayashi M, Kobayashi K, Adachi H, Yoneda K. A reinforcement learning system for swarm behaviors. In: Proceedings of IEEE World Congress on Computationl Intelligence. (WCCI/IJCNN 2008) 2008. p. 3710–5.
Kuremoto T, Obayashi M, Kobayashi K, Adachi H, Yoneda K. A neuro-fuzzy learning system for adaptive swarm behaviors dealing with continuous state space. In: Proceedings of International Conference on Intelligence Computation (ICIC 2008), vol 5227. Springer, LNAI; 2009. p. 675–83.
Kuremoto T, Obayashi M, Kobayashi K. Adaptive swarm behavior acquisition by a neuro-fuzzy system and reinforcement learning algorithm. Int J Intell Compuy Cybern. 2009;2(4):724–44.
Kuremoto T, Obayashi M, Kobayashi K. An improved internal model for swarm formation and adaptive swarm behavior acquisition. J Circuit Syst Comput. 2009;18(8):1517–31.
Sato S, Nozawa A, Ide H. Characteristics of behavior of robots with emotion model. IEEJ Trans EIS. 2004;124(7):1390–5. (in Japanese).
Kusano T, Nozawa A, Ide H. Emergent of burden sharing of robots with emotion model. IEEJ Trans EIS. 2005;125(7):1037–42. (in Japanese).
Russell JA. A circumplex model of affect. J Pers Soc Psychol. 1980;39(6):1161–78.
Larsen RJ, Diener E. Promises and problems with the circumplex model of emotion. In: Clark MS, editor. Rev. personality and social psychology: emotion, vol 13. Newbury Park, CA: Sage; 1992. p. 25–59.
Kuremoto T, Obayashi M, Kobayashi K, Feng L-B. Autonomic behaviors of swarm robots driven by emotion and curiosity. In: Li K editor, LSMS/ICSEE, vol 6630. Springer, LNBI; 2010. p. 541–7.
Oudeyer P-Y, Kaplan F. Intelligent adaptive curiosity: a source of self-development. In Proceedings of 4th International Workshop on Epigenetic Robotics; 2004. p. 12–132.
Acknowledgments
We would like to thank M. Sugino and Y. Matsusaki for their early work, and a part of this study was supported by JSPS-KAKENHI (No.20500207 and No.20500277).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kuremoto, T., Obayashi, M., Kobayashi, K. et al. An Improved Internal Model of Autonomous Robots by a Psychological Approach. Cogn Comput 3, 501–509 (2011). https://doi.org/10.1007/s12559-011-9102-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12559-011-9102-7