Abstract
Artificial neural networks (ANNs) are state-of-the-art machine learning architectures modeling neurons and their connections through weights and biases. ANNs serve as universal function approximators, meaning that a sufficiently complex neural network can learn almost any function in any dimension. This flexibility, combined with backpropagation and a learning algorithm, enables to learn unknown functions with an astonishing accuracy. This chapter introduces the basics of neural networks, backpropagation, and the learning algorithm. Additionally, activation functions and regularization are treated. The derivatives with respect to the networks’ input are also explained, as these are essential for the upcoming chapters on physics-informed neural networks and the deep energy method. Finally, an outlook on more advanced network architectures is provided.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Florian Marquardt. “Machine Learning for Physicists”. Lecture. Lecture. 2019. URL: https://www.video.uni-erlangen.de/clip/id/10611.html (visited on 03/30/2020).
Howard B. Demuth et al. Neural Network Design. 2nd. Stillwater, OK, USA: Martin Hagan, 2014. ISBN: 978-0-9717321-1-7.
Giuseppe Carleo et al. “Machine learning and the physical sciences”. In: Rev. Mod. Phys. 91.4 (Dec. 6, 2019), p. 045002. ISSN: 0034-6861, 1539-0756. DOI https://doi.org/10.1103/RevModPhys.91.045002. URL: http://arxiv.org/abs/1903.10563 (visited on 01/15/2020).
Pankaj Mehta et al. “A high-bias, low-variance introduction to Machine Learning for physicists”. In: Physics Reports 810 (May 2019), pp. 1–124. ISSN: 03701573. DOI https://doi.org/10.1016/j.physrep.2019.03.001. URL: http://arxiv.org/abs/1803.08823 (visited on 01/14/2020).
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. ISBN: 0-262-03561-8. URL: http://www.deeplearningbook.org.
G. Cybenko. “Approximation by superpositions of a sigmoidal function”. In: Math. Control Signal Systems 2.4 (Dec. 1, 1989), pp. 303–314. ISSN: 1435-568X. DOI https://doi.org/10.1007/BF02551274 (visited on 05/20/2020).
Michael A. Nielsen. Neural Networks and Deep Learning. Determination Press, 2015. URL: http://neuralnetworksanddeeplearning.com (visited on 03/13/2020).
Hrushikesh Mhaskar, Qianli Liao, and Tomaso Poggio. “Learning Functions: When Is Deep Better Than Shallow”. In: arXiv:1603.00988 [cs] (May 29, 2016) (visited on 05/20/2020).
Edwin Hewitt and Robert E. Hewitt. “The Gibbs-Wilbraham phenomenon: An episode in fourier analysis”. In: Arch. Hist. Exact Sci. 21.2 (1979), pp. 129–160. ISSN: 0003-9519, 1432-0657. DOI https://doi.org/10.1007/BF00330404 (visited on 11/05/2020).
B. Llanas, S. Lantarón, and F. J. Sáinz. “Constructive Approximation of Discontinuous Functions by Neural Networks”. In: Neural Process Lett 27.3 (June 2008), pp. 209–226. ISSN: 1370-4621, 1573-773X. DOI https://doi.org/10.1007/s11063-007-9070-9 (visited on 05/20/2020).
Florian Marquardt. Visualize the output of a multilayer network. 2017. URL: http://www.thp2.nat.uni-erlangen.de/images/0/0e/MultiLayerNet_SimpleExample.py (visited on 07/29/2020).
Atılım Gunes Baydin et al. “Automatic Differentiation in Machine Learning: a Survey”. In: (2018), p. 43.
Christopher Olah. Calculus on Computational Graphs: Backpropagation. colah’s blog. Aug. 31, 2015. URL: http://colah.github.io/posts/2015-08-Backprop/ (visited on 10/01/2020).
Andrew Ng. “Machine Learning”. Online course. Online course. 2020. URL: https://www.coursera.org/learn/machine-learning?.
François Chollet. Deep learning with Python. OCLC: ocn982650571. Shelter Island, New York: Manning Publications Co, 2018. 361 pp. ISBN: 978-1-61729-443-3.
D. Randall Wilson and Tony R. Martinez. “The general inefficiency of batch training for gradient descent learning”. In: Neural Networks 16.10 (Dec. 2003), pp. 1429–1451. ISSN: 08936080. DOI https://doi.org/10.1016/S0893-6080(03)00138-2 (visited on 07/11/2020).
Adam Paszke et al. “PyTorch: An Imperative Style, High-Performance Deep Learning Library”. In: arXiv:1912.01703 [cs, stat] (Dec. 3, 2019) (visited on 09/29/2020).
PyTorch Documentation. Linear — PyTorch 1.6.0 documentation. 2020. URL: https://pytorch.org/docs/ (visited on 09/29/2020).
Kian Katanforoosh and Daniel Kunin. Initializing neural networks. deeplearning.ai. 2019. URL: https://www.deeplearning.ai/ai-notes/initialization/ (visited on 12/22/2020).
Y. Lecun et al. “Gradient-based learning applied to document recognition”. In: Proceedings of the IEEE 86.11 (Nov. 1998). Conference Name: Proceedings of the IEEE, pp. 2278–2324. ISSN: 1558-2256. DOI https://doi.org/10.1109/5.726791.
Christopher Olah. Understanding LSTM Networks. colah’s blog. Aug. 27, 2015. URL: http://colah.github.io/posts/2015-08-Understanding-LSTMs/ (visited on 04/29/2020).
Sepp Hochreiter. “Untersuchungen zu dynamischen neuronalen Netzen”. In: Diploma, Technische Universität München 91.1 (1991).
Author information
Authors and Affiliations
Corresponding author
3.1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Kollmannsberger, S., D’Angella, D., Jokeit, M., Herrmann, L. (2021). Neural Networks. In: Deep Learning in Computational Mechanics. Studies in Computational Intelligence, vol 977. Springer, Cham. https://doi.org/10.1007/978-3-030-76587-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-76587-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-76586-6
Online ISBN: 978-3-030-76587-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)