Abstract
Hybrid electric vehicles (HEVs) are acknowledged to be an effective way to improve the efficiency of internal combustion engines (ICEs) and reduce fuel consumption. Although the ICE in an HEV can maintain high efficiency during driving, its thermal efficiency is approximately 40%, and the rest of the fuel energy is discharged through different kinds of waste heat. Therefore, it is important to recover the engine waste heat. Because of the great waste heat recovery performance of the organic Rankine cycle (ORC), an HEV integrated with an ORC (HEV-ORC) has been proposed. However, the addition of ORC creates a stiff and multi-energy problem, greatly increasing the complexity of the energy management system (EMS). Considering the great potential of deep reinforcement learning (DRL) for solving complex control problems, this work proposes a DRL-based EMS for an HEV-ORC. The simulation results demonstrate that the DRL-based EMS can save 2% more fuel energy than the rule-based EMS because the former provides higher average efficiencies for both engine and motor, as well as more stable ORC power and battery state. Furthermore, the battery always has sufficient capacity to store the ORC power. Consequently, DRL showed great potential for solving complex energy management problems.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
The World Bank. CO2 emissions from transport (% of total fuel combustion). https://data.worldbank.org/indicator/EN.CO2.TRAN.ZS
EIA. International Energy Outlook. 2016. https://www.eia.gov/out-looks/aeo/er/
Di Battista D, Fatigati F, Carapellucci R, et al. Inverted Brayton Cycle for waste heat recovery in reciprocating internal combustion engines. Appl Energy, 2019, 253: 113565
The ICCCT. CO2 Emission Standards for Passenger Cars and Light-Commercial Vehicles in the European Union. Technical Report. January 2019
Faraj M, Basir O. Range anxiety reduction in battery-powered vehicles. In: 2016 IEEE Transportation Electrification Conference and Expo (ITEC). Dearborn: IEEE, 2016
Abedin M J, Masjuki H H, Kalam M A, et al. Energy balance of internal combustion engines using alternative fuels. Renew Sustain Energy Rev, 2013, 26: 20–33
Yu G, Shu G, Tian H, et al. Multi-approach evaluations of a cascade-Organic Rankine Cycle (C-ORC) system driven by diesel engine waste heat: Part B-techno-economic evaluations. Energy Convers Manage, 2016, 108: 596–608
Algieri A, Morrone P. Comparative energetic analysis of high-temperature subcritical and transcritical Organic Rankine Cycle (ORC). A biomass application in the Sibari district. Appl Thermal Eng, 2012, 36: 236–244
Shu G, Wang X, Tian H. Theoretical analysis and comparison of Rankine cycle and different organic Rankine cycles as waste heat recovery system for a large gaseous fuel internal combustion engine. Appl Thermal Eng, 2016, 108: 525–537
Biswas A, Emadi A. Energy management systems for electrified powertrains: State-of-the-art review and future trends. IEEE Trans Veh Technol, 2019, 68: 6453–6467
Lee H D, Sul S K. Fuzzy-logic-based torque control strategy for parallel-type hybrid electric vehicle. IEEE Trans Ind Electron, 1998, 45: 625–632
Piccolo A, Ippolito L, Galdi V, et al. Optimisation of energy flow management in hybrid electric vehicles via genetic algorithms. In: ASME International Conference on Advanced Intelligent Mechatronics. Como: IEEE, 2001
Pérez L V, Bossio G R, Moitre D, et al. Optimization of power management in an hybrid electric vehicle using dynamic programming. Math Comput Simul, 2006, 73: 244–254
Zhang S, Xiong R, Sun F. Model predictive control for power management in a plug-in hybrid electric vehicle with a hybrid energy storage system. Appl Energy, 2017, 185: 1654–1662
Xu B, Rathod D, Zhang D, et al. Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle. Appl Energy, 2020, 259: 114200
Feru E, Murgovski N, de Jager B, et al. Supervisory control of a heavy-duty diesel engine with an electrified waste heat recovery system. Control Eng Practice, 2016, 54: 190–201
Mansour C, Bou Nader W, Dumand C, et al. Waste heat recovery from engine coolant on mild hybrid vehicle using organic Rankine cycle. Proc Inst Mech Eng D-J Autom Eng, 2019, 233: 2502–2517
Kruijt K, Verbruggen F J R, Speetjens M F M, et al. Modeling and control of a waste heat recovery system for integrated powertrain design optimization. IFAC-PapersOnLine, 2019, 52: 598–603
Zhao C, Yin H, Yang Z, et al. Equivalent series resistance-based energy loss analysis of a battery semiactive hybrid energy storage system. IEEE Trans Energy Convers, 2015, 30: 1081–1091
Zhang Q, Ju F, Zhang S, et al. Power management for hybrid energy storage system of electric vehicles considering inaccurate terrain information. IEEE Trans Automat Sci Eng, 2017, 14: 608–618
Wieczorek M, Lewandowski M. A mathematical representation of an energy management strategy for hybrid energy storage system in electric vehicle and real time optimization using a genetic algorithm. Appl Energy, 2017, 192: 222–233
Buşoniu L, de Bruin T, Tolić D, et al. Reinforcement learning for control: Performance, stability, and deep approximators. Annu Rev Control, 2018, 46: 8–28
Zhang B, Hu W, Cao D, et al. Deep reinforcement learning-based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy. Energy Convers Manage, 2019, 202: 112199
Silver D, Schrittwieser J, Simonyan K, et al. Mastering the game of Go without human knowledge. Nature, 2017, 550: 354–359
Tan H, Zhang H, Peng J, et al. Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space. Energy Convers Manage, 2019, 195: 548–560
Xiong R, Cao J, Yu Q. Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle. Appl Energy, 2018, 211: 538–548
Zou Y, Liu T, Liu D, et al. Reinforcement learning-based real-time energy management for a hybrid tracked vehicle. Appl Energy, 2016, 171: 372–382
Zou R, Fan L, Dong Y, et al. DQL energy management: An online-updated algorithm and its application in fix-line hybrid electric vehicle. Energy, 2021, 225: 120174
Lian R, Peng J, Wu Y, et al. Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle. Energy, 2020, 197: 117297
Wang X, Shu G, Tian H, et al. Dynamic analysis of the dual-loop Organic Rankine Cycle for waste heat recovery of a natural gas engine. Energy Convers Manage, 2017, 148: 724–736
Zhou M, Zhang Y, Wang X. Modeling and simulation of power assembly for signal-axle parallel hybrid electric vehicles. Electric Mach Control, 2019, 13: 36–40
Shu G, Li X, Tian H, et al. Alkanes as working fluids for high-temperature exhaust heat recovery of diesel engine using organic Rankine cycle. Appl Energy, 2014, 119: 204–217
Mikielewicz D, Mikielewicz J. A thermodynamic criterion for selection of working fluid for subcritical and supercritical domestic micro CHP. Appl Thermal Eng, 2010, 30: 2357–2362
Wang X, Shu G, Tian H, et al. Effect factors of part-load performance for various Organic Rankine cycles using in engine waste heat recovery. Energy Convers Manage, 2018, 174: 504–515
Gao Y, Wang X, Tian H, et al. Quantitative analysis of fuel-saving potential for waste heat recovery system integrated with hybrid electric vehicle. Int J Energy Res, 2020, 44: 11152–11170
Liu T, Wang E, Meng F, et al. Operation characteristics and transient simulation of an ICE-ORC combined system. Appl Sci, 2019, 9: 1639
Wang X, Shu G, Tian H, et al. Engine working condition effects on the dynamic response of organic Rankine cycle as exhaust waste heat recovery system. Appl Thermal Eng, 2017, 123: 670–681
Jensen J, Tummescheit H. Moving boundary models for dynamic simulation of two-phase flows. In: The 2nd International Modelica Conference. Oberpfaffenhofen, 2002. 18–19
Johan P, Paolino T, Olivier L. Improving the control performance of an organic Rankine cycle system for waste heat recovery from a heavy-duty diesel engine using a model-based approach. In: 52nd IEEE Conference on Decision and Control. Firenze, 2013. 10–13
Qi X, Luo Y, Wu G, et al. Deep reinforcement learning enabled self-learning control for energy efficient driving. Transp Res Part C-Emerg Tech, 2019, 99: 67–81
Pisu P, Rizzoni G. A comparative study of supervisory control strategies for hybrid electric vehicles. IEEE Trans Contr Syst Technol, 2007, 15: 506–518
Li Y, He H, Khajepour A, et al. Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information. Appl Energy, 2019, 255: 113762
Wang X, Wang R, Jin M, et al. Control of superheat of organic Rankine cycle under transient heat source based on deep reinforcement learning. Appl Energy, 2020, 278: 115637
Author information
Authors and Affiliations
Corresponding authors
Additional information
This work was supported by the National Natural Science Foundation of China (Grant No. 51906173).
Rights and permissions
About this article
Cite this article
Wang, X., Wang, R., Shu, G. et al. Energy management strategy for hybrid electric vehicle integrated with waste heat recovery system based on deep reinforcement learning. Sci. China Technol. Sci. 65, 713–725 (2022). https://doi.org/10.1007/s11431-021-1921-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11431-021-1921-0