Abstract
This chapter presents an introduction to recurrent neural networks for readers familiar with artificial neural networks in general, and multi-layer perceptrons trained with gradient descent algorithms (back-propagation) in particular. A recurrent neural network (RNN) is an artificial neural network with internal loops. These internal loops induce recursive dynamics in the networks and thus introduce delayed activation dependencies across the processing elements (PEs) in the network.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Allen, R.B., Alspector, J.: Learning of stable states in stochastic asymmetric networks. Technical Report TM-ARH-015240, Bell Communications Research, Morristown, NJ (1989)
Atiya, A.F.: Learning on a general network. In: Neural Information Processing Systems, New York, pp. 22–30 (1988)
Back, A.D., Tsoi, A.C.: FIR and IIR synapses, a new neural network architecture for time series modeling. Neural Computation 3, 375–385 (1991)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gadient descent is difficult. IEEE Transactions on Neural Networks 5, 157–166 (1994)
Chen, L., Chua, H., Tan, P.: Grammatical inference using an adaptive recurrent neural network. Neural Processing Letters 8, 211–219 (1998)
Chen, S., Billings, S., Grant, P.: Nonlinear system identification using neural networks. International Journal of Control 51(6), 1191–1214 (1990)
Cohen, M.A., Grossberg, S.: Stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man and Cybernetics 13, 815–826 (1983)
Elman, J.L.: Finding structure in time. Cognitive Science 14, 179–211 (1990)
Elman, J.L.: Distributed representations, simple recurrent networks and grammatical structure. Machine Learning 7, 195–225 (1991)
Fahlman, S.E., Lebiere, C.: The cascade-correlation learning architecture. Technical Report CMU-CS-90-100, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA (February 1990)
Forcada, M.L., Ñeco, R.P.: Recursive Hetero-Associative Memories for Translation. In: Mira, J., Moreno-Díaz, R., Cabestany, J. (eds.) IWANN 1997. LNCS, vol. 1240, pp. 453–462. Springer, Heidelberg (1997)
Frasconi, P., Gori, M., Soda, G.: Local feedback multilayered networks. Neural Computation 4, 120–130 (1992)
Galland, C.C., Hinton, G.E.: Deterministic Boltzman learning in networks with asymmetric connectivity. Technical Report CRG-TR-89-6, University of Toronto Department of Computer Science (1989)
Ge, H., Du, W., Qian, F., Liang, Y.: Identification and control of nonlinear systems by a time-delay recurrent neural network. Neurocomputing 72, 2857–2864 (2009)
Giles, C., Kuhn, G., Williams, R.: Dynamic recurrent neural networks: theory and applications. IEEE Trans. Neural Netw. 5(2), 153–156 (1994)
Giles, C.L., Chen, D., Miller, C.B., Chen, H.H., Sun, G.Z., Lee, Y.C.: Second-order recurrent neural networks for grammatical inference. In: 1991 IEEE INNS International Joint Conference on Neural Networks, Seattle, Piscataway, NJ, vol. 2, pp. 271–281. IEEE Press (1991)
Giles, C.L., Horne, B.G., Lin, T.: Learning a class of large finite state machines with a recurrent neural network. Neural Networks 8, 1359–1365 (1995)
Giles, C.L., Miller, C.B., Chen, D., Chen, H.H., Sun, G.Z., Lee, Y.C.: Learning and extracting finite state automata with second-order recurrent neural networks. Neural Computation 4, 395–405 (1992)
Gori, M., Bengio, Y., Mori, R.D.: Bps: A learning algorithm for capturing the dynamic nature of speech. In: International Joint Conference on Neural Networks, vol. II, pp. 417–423 (1989)
Harigopal, U., Chen, H.C.: Grammatical inference using higher order recurrent neural networks. In: Proceedings of the Twenty-Fifth Southeastern Symposium on System Theory, SSST 1993, pp. 338–342 (1993)
Hinton, T.J., abd Sejnowski, G.E.: Optimal perceptual inference. In: Proceedines of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 448–453. IEEE Computer Society (1983)
Hochreiter, S.: Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis (1991)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9, 1735–1780 (1997)
Hopcroft, J.E., Ullman, J.D.: Introduction to Automata Theory, Languages, and Computation. Addison-Wesley (1979)
Jordan, M.I.: Supervised learning and systems with excess degrees of freedom. Technical Report COINS Technical Report 88–27, Massachusetts Institute of Technology (1988)
Karakasoglu, A., Sudharsanan, S., Sundareshan, M.K.: Identification and decentralized adaptive control using dynamic neural networks with application to robotic manipulators. IEEE Trans. Neural Networks 4, 919–930 (1993)
Karray, F.O., Silva, C.: Soft Computing and Intelligent Systems Design. Addison Wesley (2004)
Kilian, J., Siegelmann, H.T.: On the power of sigmoid neural networks. In: Proceedings of the Sixth ACM Workshop on Computational Learning Theory, pp. 137–143. ACM Press (1993)
Kolen, J.F., Kremer, S.C. (eds.): A Field Guide to Dynamical Recurrent Networks. Wiley-IEEE Press (2001)
Kuo, J., Celebi, S.: Adaptation of memory depth in the gamma filter. In: Acoustics, Speech and Signal Processing IEEE Conference, pp. 1–4 (1994)
Kuroe, Y.: Representation and Identification of Finite State Automata by Recurrent Neural Networks. In: Pal, N.R., Kasabov, N., Mudi, R.K., Pal, S., Parui, S.K. (eds.) ICONIP 2004. LNCS, vol. 3316, pp. 261–268. Springer, Heidelberg (2004)
Lippmann, R.P.: An introduction to computing with neural nets. IEEE ASSP Magazine 4, 4–22 (1987)
Mozer, M.: A focused background algorithm for temporal pattern recognition. Complex Systems 3 (1989)
Mozer, M.C.: Induction of multiscale temporal structure. In: Advances in Neural Information Processing Systems 4, pp. 275–282. Morgan Kaufmann (1992)
Nguyen, M., Cottrell, G.: A technique for adapting to speech rate. In: Kamm, C., Kuhn, G., Yoon, B., Chellapa, R., Kung, S. (eds.) Neural Networks for Signal Processing 3. IEEE Press (1993)
Omlin, C.W., Giles, C.L.: Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM 43(6), 937–972 (1996)
Patan, K.: Locally Recurrent Neural Networks. In: Patan, K. (ed.) Artificial. Neural Net. for the Model. & Fault Diagnosis. LNCIS, vol. 377, pp. 29–63. Springer, Heidelberg (2008)
Pollack, J.B.: On Connectionist Models of Natural Language Processing. PhD thesis, Computer Science Department of the University of Illinois at Urbana-Champaign, Urbana, Illinois, Available as TR MCCS-87-100, Computing Research Laboratory, New Mexico State University, Las Cruces, NM (1987)
Principe, J.C., de Vries, B., de Oliveira, P.G.: The gamma filter - a new class of adaptive IIR filter with restricted feedback. IEEE Transactions on Signal Processing 41, 649–656 (1993)
Renals, S., Rohwer, R.: A study of network dynamics. Journal of Statistical Physics 58, 825–848 (1990)
Robinson, A.J.: Dynamic Error Propagation Networks. Ph.d., Cambridge University Engineering Department (1989)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Parallel Distributed Processing. MIT Press, Cambridge (1986)
Schmidhuber, J.H.: A fixed size storage o(n 3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation 4(2), 243–248 (1992)
Sejnowski, T.J., Rosenberg, C.R.: Parallel networks that learn to pronounce english text. Complex Syst. I, 145–168 (1987)
Shannon, C.E.: Communication in the presence of noise. Proc. Institute of Radio Engineers 37(1), 10–21 (1949); reprinted as classic paper in: Proc. IEEE 86(2) (February 1998)
Shearer, J.L., Murphy, A.T., Richardson, H.H.: Introduction to System Dynamics. Addison-Wesley, Reading (1971)
Siegelmann, H.T., Sontag, E.D.: Turing computability with neural nets. Applied Mathematics Letters 4(6), 77–80 (1991)
Silva, T.O.: Laguerre filters - an introduction. Revista do Detua 1(3) (1995)
Smith, J.O.: Delay lines. Physical Audio Signal Processing (2010), http://ccrma.stanford.edu/~jos/pasp/Tapped_Delay_Line_TDL.htm (cited November 28, 2010)
Smith, S.W.: The scientist and engineer’s guide to digital signal processing. California Technical Publishing (2006), http://www.dspguide.com/ch15.htm (cited November 29, 2010)
Tsoi, A.C., Back, A.D.: Locally recurrent globally feedforward networks: A critical review of architectures. IEEE Transactions on Neural Networks 5, 229–239 (1994)
Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, L.: Phonemic recognition using time delay neural networks. IEEE Trans. Acoustic Speech and Signal Processing 37(3), 328–339 (1989)
Werbos, P.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioural Sciences. Phd thesis, Harvard University (1974)
Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1, 270–289 (1989)
Won, S.H., Song, I., Lee, S.Y., Park, C.H.: Identification of finite state automata with a class of recurrent neural networks. IEEE Transactions on Neural Networks 21(9), 1408–1421 (2010)
Yan, P.F., Zhang, C.S.: Artificial Neural Network and Simulated Evolutionary Computation. Thinghua University Press, Beijing (2000)
Zamarreno, J.M., Vega, P.: State space neural network. Properties and application. Neural Networks 11, 1099–1112 (1998)
Zeng, Z., Goodman, R.M., Smyth, P.: Learning finite state machines with self-clustering recurrent networks. Neural Computation 5(6), 977–990 (1993)
Zeng, Z., Goodman, R.M., Smyth, P.: Discrete recurrent neural networks for grammatical inference. IEEE Transactions on Neural Networks 5(2), 320–330 (1994)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Marhon, S.A., Cameron, C.J.F., Kremer, S.C. (2013). Recurrent Neural Networks. In: Bianchini, M., Maggini, M., Jain, L. (eds) Handbook on Neural Information Processing. Intelligent Systems Reference Library, vol 49. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36657-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-36657-4_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-36656-7
Online ISBN: 978-3-642-36657-4
eBook Packages: EngineeringEngineering (R0)