Abstract
In recent years, artificial neural networks (ANNs) and deep learning have become increasingly popular across a wide range of scientific and technical fields, including fluid mechanics. While it will take time to fully grasp the potentialities as well as the limitations of these methods, evidence is starting to accumulate that point to their potential in helping solve problems for which no theoretically optimal solution method is known. This is particularly true in fluid mechanics, where problems involving optimal control and optimal design are involved. Indeed, such problems are famously difficult to solve effectively with traditional methods due to the combination of non linearity, non convexity, and high dimensionality they involve. By contrast, deep reinforcement learning (DRL), a method of optimization based on teaching empirical strategies to an ANN through trial and error, is well adapted to solving such problems. In this short review, we offer an insight into the current state of the art of the use of DRL within fluid mechanics, focusing on control and optimal design problems.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Krizhevsky A., Sutskever I., Hinton G. E. Imagenet classification with deep convolutional neural networks [C]. Advances in Neural Information Processing Systems, Lake Tahoe, USA, 2012, 1097–1105.
He K., Zhang X., Ren S. et al. Deep residual learning for image recognition [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016, 770–778.
Rabault J., Kolaas J., Jensen A. Performing particle image velocimetry using artificial neural networks: A proof-of-concept [J]. Measurement Science and Technology, 2017, 28(12): 125301.
Kober J., Bagnell J. A., Peters J. Reinforcement learning in robotics: A survey [J]. The International Journal of Robotics Research, 2013, 32(11): 1238–1274.
Gu S., Holly E., Lillicrap T. et al. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates [C]. 2017 IEEE international conference on robotics and automation (ICRA), IEEE, Singapore, 2017, 3389–3396.
Schmidhuber J. Deep learning in neural networks: An overview [J]. Neural networks, 2015, 61: 85–117.
Lillicrap T. P., Hunt J. J., Pritzel A. et al. Continuous control with deep reinforcement learning [EB/OL]. arXiv preprint, 2015, arXiv:1509.02971.
Sutton R. S., Barto A. G. Reinforcement learning: An introduction [M]. Cambridge, USA: MIT press, 2018.
Rabault J., Zhang W., Xu H. Deep reinforcement learning in fluid mechanics: a promising method in both active flow control and shape optimization [C]. International Symposium on High Fidelity Computational Methods and Applications, Shanghai, China, 20129.
Rabault J. Deep reinforcement learning applied to fluid mechanics: materials from the 2019 flow/interface school on machine learning and data driven methods [C]. International Symposium on High Fidelity Computational Methods and Applications, Shanghai, China, 2019.
Rosenblatt F. The perceptron: A perceiving and recognizing automation [R]. New York, USA: Cornell Aeronautical Laboratory, 1957, Report 85-60-1.
Goodfellow I., Bengio Y., Courville A. 2017 The Deep Learning Book [M]. Cambridge, USA: MIT Press.
LeCun Y., Bengio Y., Hinton G. Deep learning [J]. Nature, 2015, 521(7553): 436–444.
LeCun Y., Bengio Y. Convolutional networks for images, speech, and time series (The handbook of brain theory and neural networks) [M]. Cambridge, USA: MIT Press, 1998.
Hornik K., Stinchcombe M., White H. Multilayer feedforward networks are universal approximators [J]. Neural Networks, 1989, 2(5): 359–366.
Glorot X., Bordes A., Bengio Y. Deep sparse rectifier neural networks [C]: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Rome, Italy, 2011, 315–323.
Glorot X., Bengio Y. Understanding the difficulty of training deep feedforward neural networks [C]. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Chia Laguna Resort, Italy, 2010, 249–256.
Srivastava N., Hinton G., Krizhevsky A. et al. Dropout: A simple way to prevent neural networks from overfitting [J]. The Journal of Machine Learning Research, 2014, 15(1): 1929–1958.
Ioffe S., Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift [EB/OL]. arXiv preprint, 2015, arXiv:1502.03167.
He K., Zhang X., Ren S. et al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification [C]. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 2015, 1026–1034.
Li X., Chen H., Qi X. et al. H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes [J]. IEEE Transactions on Medical Imaging, 2018, 37(12): 2663–2674.
Kingma D. P., Ba J. Adam: A method for stochastic optimization [EB/OL]. arXiv preprint, 2014, arXiv: 1412.6980.
Duchi J., Hazan E., Singer Y. Adaptive subgradient methods for online learning and stochastic optimization [J]. Journal of Machine Learning Research, 2011, 12(7): 2121–2159.
Abadi M., Barham P., Chen J. et al. Tensorflow: A system for large-scale machine learning [C]. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16), Savannah, USA, 2016, 265–283.
Paszke A., Gross S., Massa F. et al. PyTorch: An imperative style, high-performance deep learning library [C]. Advances in Neural Information Processing Systems, Vancouver, Canada, 2019, 8024–8035.
Silver D., Schrittwieser J., Simonyan K. et al. Mastering the game of go without human knowledge [J]. Nature, 2017, 550(7676): 354–359.
Silver D., Hubert T., Schrittwieser J. et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play [J]. Science, 2018, 362(6419): 1140–1144.
Knight W. Google just gave control over data center cooling to an AI [EB/OL]. 2018, https://www.technologyreview.com/s/611902/google-just-gave-control-over-data-center-cooling-to-an-ai/.
Bellman R. A Markovian decision process [J]. Journal of Mathematics and Mechanics, 1957, 6(4): 679–684.
Bellman R. E., Dreyfus S. E. Applied dynamic programming [M]. Princeton, USA: Princeton University Press, 2015.
Van Hasselt H., Guez A., Silver D. Deep reinforcement learning with double q-learning [C]. Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, USA, 2016.
Schulman J., Levine S., Abbeel P. et al. Trust region policy optimization [C]. International Conference on Machine Learning, Guangzhou, China, 2015, 1889–1897.
Garnier P., Viquerat J., Rabault J. et al. A review on deep reinforcement learning for fluid mechanics [EB/OL]. arXiv preprint, 2019, arXiv:1908.04127.
Schaul T., Quan J., Antonoglou I. et al. Prioritized experience replay [EB/OL]. arXiv preprint, 2015, arXiv:1511.05952.
Pinto L., Andrychowicz M., Welinder P. et al. Asymmetric actor critic for image-based robot learning [EB/OL]. arXiv preprint, 2017, arXiv:1710.06542.
Schulman J., Wolski F., Dhariwal P. et al. Proximal policy optimization algorithms [EB/OL]. arXiv preprint, 2017, arXiv:1707.06347,.
Achiam J, Sastry S. Surprise-based intrinsic motivation for deep reinforcement learning[J]. arXiv preprint, 2017, arXiv:1703.01732.
Savinov N., Raichuk A., Marinier R. et al. Episodic curiosity through reachability [EB/OL]. arXiv preprint, 2018, arXiv:1810.02274.
Ha D., Schmidhuber J. World models [EB/OL]. arXiv preprint, 2018, arXiv:1803.10122.
Salimans T., Chen R. Learning Montezuma’s revenge from a single demonstration [EB/OL]. arXiv preprint, 2018, arXiv:1812.03381.
Tensorforce: A tensorflow library for applied reinforcement learning [EB/OL]. 2017, https://tensorforce.readthedocs.io/en/latest/.
Stable baselines [EB/OL]. 2018, https://github.com/hill-a/stable-baselines.
Duriez T., Brunton S. L., Noack B. R. Machine learning control-taming nonlinear dynamics and turbulence [M]. Cham, Switzerland: Springer International Publishing, 2017.
Novati G., Verma S., Alexeev D. et al. Synchronised swimming of two fish [J]. Bioinspiration and Biomimetics, 2017, 12(3): 036001.
Reddy G., Celani A., Sejnowski T. J. et al. Learning to soar in turbulent environments [J]. Proceedings of the National Academy of Sciences, 2016, 113(33): E4877–E4884.
Bøhn E., Coates E. M., Moe S. et al. Deep reinforcement learning attitude control of fixed-wing UAVs using proximal policy optimization [C]. 2019 International Conference on Unmanned Aircraft Systems (ICUAS), IEEE, Atlanta, USA, 2019, 523–533.
Hwangbo J., Sa I., Siegwart R. et al. Control of a quadrotor with reinforcement learning [J]. IEEE Robotics and Automation Letters, 2017, 2(4): 2096–2103.
Biferale L., Bonaccorso F., Buzzicotti M. et al. Zermelo’s problem: Optimal point-to-point navigation in 2D turbulent flows using reinforcement learning [J]. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2019, 29(10): 103138.
Rabault J., Kuchta M., Jensen A. et al. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control [J]. Journal of Fluid Mechanics, 2019, 865: 281–302.
Ren F., Rabault J., Tang H. Active flow control of flow past a circular cylinder at moderate reynolds number using deep reinforcement learning [C]. International Symposium on High Fidelity Computational Methods and Applications, Shanghai, China, 2019.
Rabault J., Kuhnle A. Accelerating deep reinforcement learning strategies of flow control through a multienvironment approach [J]. Physics of Fluids, 2019, 31(9): 094105.
Clark J., Amodei D. Faulty reward functions in the wild [EB/OL]. 2016, https://openai.com/blog/faulty-reward-functions/.
Baker B., Ingmar K., Markov T. et al. Emergent tool use from multi-agent interaction [EB/OL]. arXiv preprint, 2019, arXiv:1909.07528.
Belus V., Rabault J., Viquerat J. et al. Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film [J]. AIP Advances, 2019, 9(12): 125014.
Rabault J., Belus V., Viquerat J. et al. Exploiting locality and physical invariants to design effective deep reinforcement learning control of the unstable falling liquid film [C]. The 1st Graduate Forum of CSAA and the 7th International Academic Conference for Graduates, NUAA, Nanjing, China, 2019.
Bucci M. A., Semeraro O., Allauzen A. et al. Control of chaotic systems by deep reinforcement learning [EB/OL]. arXiv preprint, 2019, arXiv:1906.07672.
Corbetta A., Beintema G., Biferale L. et al. Reinforcement learning versus linear control of Rayleigh-Bénard convection [C]. American Physical Society, Division of Fluid Dynamics Meeting, Philadelphia, USA, 1998.
Collis S. S., Joslin R. D., Seifert A. et al. Issues in active flow control: Theory, control, simulation, and experiment [J]. Progress in Aerospace Sciences, 2004, 40(4–5): 237–289.
Cattafesta III L. N., Sheplak M. Actuators for active flow control [J]. Annual Review of Fluid Mechanics, 2011, 43: 247–272.
Hwangbo J., Lee J., Dosovitskiy A. et al. Learning agile and dynamic motor skills for legged robots [EB/OL]. arXiv preprint, 2019, arXiv:1901.08652.
Yan X., Zhu J., Kuang M. et al. Aerodynamic shape optimization using a novel optimizer based on machine learning techniques [J]. Aerospace Science and Technology, 2019, 86: 826–835.
Viquerat J., Rabault J., Kuhnle A. et al. Direct shape optimization through deep reinforcement learning [EB/OL]. arXiv preprint, 2019, arXiv:1908.09885.
Acknowledgements
This work was supported by the National Numerical Wind Tunnel Project (Grant No. NNW2019ZT4-B09), the National Natural Science Foundation of China (Grant Nos. 91852106, 91841303).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Rabault, J., Ren, F., Zhang, W. et al. Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization. J Hydrodyn 32, 234–246 (2020). https://doi.org/10.1007/s42241-020-0028-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42241-020-0028-y