Abstract
The synthesis of reinforcement learning techniques, as adaptive actor-critic method, has provided sound techniques to learn a robot’s action, generally using recurrent neural networks that approximate the unknown functionals without prior training. On the other hand, invariant manifolds have been studied to yield dimensionality reduction for efficient learning. In this paper, inspired by both schemes and the seminal approach of Campos and Lewis, we propose an integral manifold as a dynamical performance evaluator, PE, to drive an adaptive critic-action scheme. It is shown that an integral sliding mode is enforced; thus, the learning is driven by an invariant manifold. Thus, an exponential error convergence with closed-loop stability in the Lyapunov sense is guarantee together with smooth control action. Simulations show the numerical behavior for a nonlinear dynamical robot learning to follow a time varying trajectory.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Interested reader is referred to [7] for further explanation.
References
Lewis, F.W., Jagannathan, S., Yesildirak, A.: Neural Network Control of Robot Manipulators and Non-Linear Systems. CRC Press, Boca Raton (1998)
Lesort, T., Diaz-Rodriguez, N., Goudou, J.-F., Filliat, D.: State representation learning for control: an overview. Neural Netw. 108, 379–392 (2018)
Scharf, F., Helfenstein, F., Jager, J.: Actor vs critic: learning the policy or learning the value. In: Belousov, B., Abdulsamad, H., Klink, P., Parisi, S., Peters, J. (eds.) Reinforcement Learning Algorithms: Analysis and Applications, vol. 883, pp. 123–134. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-41188-6_11
Campos, J., Lewis, F.L.: Adaptive critic neural network for feedforward compensation. In: Proceedings of American Control Conference, pp. 2813–2818 (1999)
Bhasin, S., Sharma, N., Patre, P.M., Dixon, W.E.: Robust asymptotic tracking of a class of nonlinear systems using an adaptive critic based controller. In: Proceedings of American Control Conference, pp. 3223–3228 (2010)
Lewis, F.L., Vrabie, D., Vamvoudakis, K.G.: Reinforcement learning and feedback control: using natural decision methods to design optimal adaptive controllers. IEEE Control Syst. Mag. 32(6), 76–105 (2012)
Parra-Vega, V., Arimoto, S., Liu, Y.H., Hirzinger, G., Akella, P.: Dynamic sliding PID control for tracking of robot manipulators: theory and experiments. IEEE Trans. Robot. Autom. 19, 967–976 (2003)
Yang, L., Enns R., Wang, Y.T., Si, J.: Direct neural dynamic programming. In: Liu, D., Antsaklis, P.J. (eds.) Stability and Control of Dynamical Systems with Applications. Control Engineering. Birkhäuser, Boston (2003)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Si, J., Yang, L., Lu, C., Sun, J., Mei, S.: Approximate dynamic programming for continuous state and control problems. In: 17th Mediterranean Conference on Control and Automation, pp. 1415–1420 (2009). https://doi.org/10.1109/MED.2009.5164745.
Barto, A.G., Sutton, R.S., Anderson, C.W.: Neuronlike elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 13(5), 835–846 (1983)
Barto, A.G., Sutton, R.S., Anderson, C.W.: Looking back on the actor-critic architecture. IEEE Trans. Syst. Man Cybern. Syst. 51(1), 40–50 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pantoja-Garcia, L., Garcia-Rodriguez, R., Parra-Vega, V. (2022). Adaptive Actor-Critic with Integral Sliding Manifold for Learning Control of Robots. In: Moreno, H.A., Carrera, I.G., Ramírez-Mendoza, R.A., Baca, J., Banfield, I.A. (eds) Advances in Automation and Robotics Research. LACAR 2021. Lecture Notes in Networks and Systems, vol 347. Springer, Cham. https://doi.org/10.1007/978-3-030-90033-5_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-90033-5_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90032-8
Online ISBN: 978-3-030-90033-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)