Abstract
The modern complicated manufacturing industry and smart manufacturing tendency have imposed new requirements on the scheduling method, such as self-regulation and self-learning capabilities. While traditional scheduling methods cannot meet these needs due to their rigidity. Self-learning is an inherent ability of reinforcement learning (RL) algorithm inhered from its continuous learning and trial-and-error characteristics. Self-regulation of scheduling could be enabled by the emerging digital twin (DT) technology because of its virtual-real mapping and mutual control characteristics. This paper proposed a DT-enabled adaptive scheduling based on the improved proximal policy optimization RL algorithm, which was called explicit exploration and asynchronous update proximal policy optimization algorithm (E2APPO). Firstly, the DT-enabled scheduling system framework was designed to enhance the interaction between the virtual and the physical job shops, strengthening the self-regulation of the scheduling model. Secondly, an innovative action selection strategy and an asynchronous update mechanism were proposed to improve the optimization algorithm to strengthen the self-learning ability of the scheduling model. Lastly, the proposed scheduling model was extensively tested in comparison with heuristic and meta-heuristic algorithms, such as well-known scheduling rules and genetic algorithms, as well as other existing scheduling methods based on reinforcement learning. The comparisons have proved both the effectiveness and advancement of the proposed DT-enabled adaptive scheduling strategy.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Błażewicz J, Domschke W, Pesch E. The job shop scheduling problem: Conventional and new solution techniques. Eur J Operational Res, 1996, 93: 1–33
Fisher M L. Optimal solution of scheduling problems using Lagrange multipliers: Part I. Oper Res, 1973, 21: 1114–1127
Lomnicki Z A. A “branch-and-bound” algorithm for the exact solution of the three-machine scheduling problem. J Operational Res Soc, 1965, 16: 89–100
Zhou H, Cheung W, Leung L C. Minimizing weighted tardiness of job-shop scheduling using a hybrid genetic algorithm. Eur J Operational Res, 2009, 194: 637–649
Salido M A, Escamilla J, Giret A, et al. A genetic algorithm for energy-efficiency in job-shop scheduling. Int J Adv Manuf Technol, 2016, 85: 1303–1314
Lin Q, Li J, Du Z, et al. A novel multi-objective particle swarm optimization with multiple search strategies. Eur J Operational Res, 2015, 247: 732–744
Liu X, Ni Z, Qiu X. Application of ant colony optimization algorithm in integrated process planning and scheduling. Int J Adv Manuf Technol, 2016, 84: 393–404
Karaboga D. Artificial bee colony algorithm. Scholarpedia, 2010, 5: 6915
Pan Q K. An effective co-evolutionary artificial bee colony algorithm for steelmaking-continuous casting scheduling. Eur J Operational Res, 2016, 250: 702–714
Qu M, Zuo Y, Xiang F, et al. An improved electromagnetism-like mechanism algorithm for energy-aware many-objective flexible job shop scheduling. Int J Adv Manuf Technol, 2022, 119: 4265–4275
Li X Y, Xie J, Ma Q J, et al. Improved gray wolf optimizer for distributed flexible job shop scheduling problem. Sci China Tech Sci, 2022, 65: 2105–2115
Wang X, Wang R, Shu G Q, et al. Energy management strategy for hybrid electric vehicle integrated with waste heat recovery system based on deep reinforcement learning. Sci China Tech Sci, 2022, 65: 713–725
Emary E, Zawbaa H M, Grosan C. Experienced gray wolf optimization through reinforcement learning and neural networks. IEEE Trans Neural Netw Learn Syst, 2018, 29: 681–694
Shahrabi J, Adibi M A, Mahootchi M. A reinforcement learning approach to parameter estimation in dynamic job shop scheduling. Comput Industrial Eng, 2017, 110: 75–82
Zhao H, Zhang C. A decomposition-based many-objective artificial bee colony algorithm with reinforcement learning. Appl Soft Computing, 2020, 86: 105879
Stricker N, Kuhnle A, Sturm R, et al. Reinforcement learning for adaptive order dispatching in the semiconductor industry. CIRP Ann, 2018, 67: 511–514
Lin C C, Deng D J, Chih Y L, et al. Smart manufacturing scheduling with edge computing using multiclass deep Q network. IEEE Trans Ind Inf, 2019, 15: 4276–4284
Shi D, Fan W, Xiao Y, et al. Intelligent scheduling of discrete automated production line via deep reinforcement learning. Int J Prod Res, 2020, 58: 1–19
Palombarini J A, Martínez E C. Automatic generation of rescheduling knowledge in socio-technical manufacturing systems using deep reinforcement learning. In: Proceedings of 2018 IEEE Biennial Congress of Argentina. San Miguel de Tucuman, 2018. 1–8
Xia K, Sacco C, Kirkpatrick M, et al. A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence. J Manuf Syst, 2021, 58: 210–230
Park I B, Huh J, Kim J, et al. A reinforcement learning approach to robust scheduling of semiconductor manufacturing facilities. IEEE Trans Autom Sci Eng, 2020, 17: 1420–1431
Zhou L, Zhang L, Horn B K P. Deep reinforcement learning-based dynamic scheduling in smart manufacturing. Procedia CIRP, 2020, 93: 383–388
Tao F, Zhang M, Nee A Y C. Digital Twin Driven Smart Manufacturing. Elsevier Academic Press, 2019
Grieves M. Digital twin: Manufacturing excellence through virtual factory replication. White Paper, 2014. 1–7
Zhang J, Deng T, Jiang H, et al. Bi-level dynamic scheduling architecture based on service unit digital twin agents. J Manuf Syst, 2021, 60: 59–79
Tao F, Zhang M, Liu Y, et al. Digital twin driven prognostics and health management for complex equipment. CIRP Ann, 2018, 67: 169–172
Schluse M, Priggemeyer M, Atorf L, et al. Experimentable digital twins—Streamlining simulation-based systems engineering for Industry 4.0. IEEE Trans Ind Inf, 2018, 14: 1722–1731
Li Y, Tao Z, Wang L, et al. Digital twin-based job shop anomaly detection and dynamic scheduling. Robotics Comput-Integrated Manuf, 2023, 79: 102443
Yan Q, Wang H, Wu F. Digital twin-enabled dynamic scheduling with preventive maintenance using a double-layer Q-learning algorithm. Comput Operations Res, 2022, 144: 105823
Zhang M, Tao F, Nee A Y C. Digital twin enhanced dynamic job-shop scheduling. J Manuf Syst, 2021, 58: 146–156
Negri E, Ardakani H D, Cattaneo L, et al. A digital twin-based scheduling framework including equipment health index and genetic algorithms. In: Proceedings of 13th International-Federation-of-Automatic-Control (IFAC) Workshop on Intelligent Manufacturing Systems (IMS). Oshawa, 2019. 52: 43–48
Pinedo M. Scheduling: Theory, Algorithms, and Systems. New York: NYU Stern School of Business, 2016. 207–214
Wang W Q, Ye C M, Tan X J. Job shop dynamic scheduling based on Q-learning algorithm. Comput Syst Appl, 2020, 29: 218–226
Adams J, Balas E, Zawack D. The shifting bottleneck procedure for job shop scheduling. Manage Sci, 1988, 34: 391–401
Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms. arXiv: 1707.06347, 2017
Girshick R. Fast R-CNN. In: Proceedings of IEEE International Conference on Computer Vision. Santiago, 2015. 1440–1448
Zoph B, Le Q V. Searching for activation functions. In: Proceedings of International Conference of Learning Representation. Vancouver, Canada, 2018. 1–13
Niu G G, Sun S D, Lafon P, et al. A decomposition approach to jobshop scheduling problem with discretely controllable processing times. Sci China Tech Sci, 2011, 54: 1240–1248
Wu Z, Pan S, Chen F, et al. A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst, 2020, 32: 4–24
Fisher H. Probabilistic learning combinations of local job-shop scheduling rules. Ind Sched, 1963, 225–251
Lawrence S. Resouce constrained project scheduling: An experimental investigation of heuristic scheduling techniques (Supplement). Dissertation for the Doctoral Degree. Pennsylvania: Carnegie-Mellon University. 1984
Applegate D, Cook W. A computational study of the job-shop scheduling problem. ORSA J Computing, 1991, 3: 149–156
Demirkol E, Mehta S, Uzsoy R. Benchmarks for shop scheduling problems. Eur J Operational Res, 1998, 109: 137–141
Taillard E. Benchmarks for basic scheduling problems. Eur J Operational Res, 1993, 64: 278–285
Zhang C, Song W, Cao Z, et al. Learning to dispatch for job shop scheduling via deep reinforcement learning. Adv Neural Inf Process Syst, 2020, 33: 1621–1632
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by the National Key R&D Program of China (Grant No. 2020YFB1713300), the Joint Open Fund of Wuhan Textile University (Grant No. KT202201005), and the Foundation of Key Laboratory of Advanced Manufacturing Technology, Ministry of Education, Guizhou University (Grant No. GZUAMT2021KF11). The authors thank for the computing support of the State Key Laboratory of Public Big Data, Guizhou University.
Rights and permissions
About this article
Cite this article
Gan, X., Zuo, Y., Zhang, A. et al. Digital twin-enabled adaptive scheduling strategy based on deep reinforcement learning. Sci. China Technol. Sci. 66, 1937–1951 (2023). https://doi.org/10.1007/s11431-022-2413-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11431-022-2413-5