Abstract
This paper aims to solve an optimal tracking control (OTC) problem of large-scale systems with multitime scales and coupled subsystems using singular perturbation (SP) theory and reinforcement learning (RL) techniques. A considerable contribution of this paper is the development of a data-driven SP-based RL method for the OTC of unknown large-scale systems with multitime scales. To achieve this, a multitime scale tracking problem was decomposed into a linear quadratic tracker problem for slow subsystems and a dynamical game problem for fast subsystems using the SP theory. Then, the distributed composite feedback controllers were found using a distributed off-policy integral RL algorithm that uses only measured data from the system in real time. Thus, the operational index can follow its prescribed target value via an approximately optimal approach. Theoretical analysis and proof are presented to demonstrate that the sum of the performances of reduced-order subsystems is approximately equal to the performance of the original large-scale system. Finally, numerical and practical examples are provided to validate the effectiveness of the proposed method.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Xie S, Huang J, Zhao C, et al. Application of neural network to hierarchical optimal control of the class of continuous time-varying large-scale systems. In: Proceedings of the IEEE International Conference on Intelligent Processing Systems, Beijing, 1997. 477–481
Bakule L. Decentralized control: an overview. Annu Rev Control, 2008, 32: 87–98
Chai T, Qin S J, Wang H. Optimal operational control for complex industrial processes. Annu Rev Control, 2014, 38: 81–92
Chai T, Ding J, Wu F. Hybrid intelligent control for optimal operation of shaft furnace roasting process. Control Eng Pract, 2011, 19: 264–275
Yuan Y, Wang Z, Guo L. Distributed quantized multi-modal fusion filtering for two-time-scale systems. Inf Sci, 2018, 432: 572–583
Chen W H, Liu Y, Zheng W X. Synchronization analysis of two-time-scale nonlinear complex networks with time-scale-dependent coupling. IEEE Trans Cybern, 2018, 49: 3255–3267
Jiang Y, Fan J, Chai T, et al. Dual-rate operational optimal control for flotation industrial process with unknown operational model. IEEE Trans Ind Electron, 2018, 66: 4587–4599
Chow J, Kokotovic P. A decomposition of near-optimum regulators for systems with slow and fast modes. IEEE Trans Automat Contr, 1976, 21: 701–705
Khalil H. Output feedback control of linear two-time-scale systems. IEEE Trans Automat Contr, 1987, 32: 784–792
Kokotovic P, Khalil H, O’Reilly J. Singular Perturbation Methods in Control: Analysis and Design. Philadelphia: Society for Industrial and Applied Mathematics, 1999
Cavallo A, de Maria G, Nistri P. Robust control design with integral action and limited rate control. IEEE Trans Automat Contr, 1999, 44: 1569–1572
Bouyekhf R, Hami A E, Moudni A E. Optimal control of a particular class of singularly perturbed nonlinear discrete-time systems. IEEE Trans Automat Contr, 2001, 46: 1097–1101
Litkouhi B, Khalil H. Multirate and composite control of two-time-scale discrete-time systems. IEEE Trans Automat Contr, 1985, 30: 645–651
Kodra K, Gajic Z. Optimal control for a new class of singularly perturbed linear systems. Automatica, 2017, 81: 203–208
Ding J, Modares H, Chai T, et al. Data-based multiobjective plant-wide performance optimization of industrial processes under dynamic environments. IEEE Trans Ind Inf, 2016, 12: 454–465
Vrabie D, Pastravanu O, Abu-Khalaf M, et al. Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica, 2009, 45: 477–484
Jiang Y, Jiang Z P. Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica, 2012, 48: 2699–2704
Xue W, Fan J, Lopez V G, et al. New methods for optimal operational control of industrial processes using reinforcement learning on two time scales. IEEE Trans Ind Inf, 2019, 16: 3085–3099
Xue W, Fan J, Lopez V G, et al. Off-policy reinforcement learning for tracking in continuous-time systems on two time scales. IEEE Trans Neural Netw Learn Syst, 2020, 32: 4334–4346
Li J, Kiumarsi B, Chai T, et al. Off-policy reinforcement learning: optimal operational control for two-time-scale industrial processes. IEEE Trans Cybern, 2017, 47: 4547–4558
Zhou L, Zhao J, Ma L, et al. Decentralized composite suboptimal control for a class of two-time-scale interconnected networks with unknown slow dynamics. Neurocomputing, 2020, 382: 71–79
Zhao J, Yang C, Dai W, et al. Reinforcement learning-based composite optimal operational control of industrial systems with multiple unit devices. IEEE Trans Ind Inf, 2021, 18: 1091–1101
Zhang L, Wang S, Wu Q, et al. Were mercury emission factors for Chinese non-ferrous metal smelters overestimated? Evidence from onsite measurements in six smelters. Environ Pollution, 2012, 171: 109–117
Li J, Ding J, Chai T, et al. Nonzero-sum game reinforcement learning for performance optimization in large-scale industrial processes. IEEE Trans Cybern, 2019, 50: 4132–4145
Saksena V, Cruz J J. Nash strategies in decentralized control of multiparameter singularly perturbed large scale systems. Large Scale Syst, 1981, 2: 219–234
Modares H, Lewis F L. Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Trans Automat Contr, 2014, 59: 3051–3056
Chen C, Xie L, Jiang Y, et al. Robust output regulation and reinforcement learning-based output tracking design for unknown linear discrete-time systems. IEEE Trans Automat Contr, 2023, 68: 2391–2398
Kiumarsi B, Lewis F L, Modares H, et al. Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics. Automatica, 2014, 50: 1167–1175
Zhang H, Cui X, Luo Y, et al. Finite-horizon H∞ tracking control for unknown nonlinear systems with saturating actuators. IEEE Trans Neural Netw Learn Syst, 2017, 29: 1200–1212
Lopez V G, Lewis F L, Wan Y, et al. Stability and robustness analysis of minmax solutions for differential graphical games. Automatica, 2020, 121: 109177
Liu M, Wan Y, Lopez V G, et al. Differential graphical game with distributed global Nash solution. IEEE Trans Control Netw Syst, 2021, 8: 1371–1382
Wang D, Hu L, Zhao M, et al. Dual event-triggered constrained control through adaptive critic for discrete-time zero-sum games. IEEE Trans Syst Man Cybern Syst, 2023, 53: 1584–1595
Li J, Chai T, Lewis F L, et al. Off-policy interleaved Q-learning: optimal control for affine nonlinear discrete-time systems. IEEE Trans Neural Netw Learn Syst, 2018, 30: 1308–1320
Wang Y Y, Shi S J, Zhang Z J. A descriptor-system approach to singular perturbation of linear regulators. IEEE Trans Automat Contr, 1988, 33: 370–373
Jiang Y, Fan J, Jia Y, et al. Data-driven flotation process operational feedback decoupling control. Acta Autom Sin, 2019, 45: 759–770
Acknowledgements
This work was supported by National Natural Science Foundation of China (Grant Nos. 62073158, 61991404, 61991400, 61673280), Science and Technology Major Project 2020 of Liaoning Province (Grant No. 2020JH1/10100008), Open Project of Key Field Alliance of Liaoning Province (Grant No. 2019KF0306), and Basic Research Project of Education Department of Liaoning Province (Grant No. LJKZ0401).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Li, J., Nie, H., Chai, T. et al. Reinforcement learning for optimal tracking of large-scale systems with multitime scales. Sci. China Inf. Sci. 66, 170201 (2023). https://doi.org/10.1007/s11432-022-3796-2
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11432-022-3796-2