Abstract
As the complexity in the structure of natural language increases, the input, output, and processing for a computer system become more challenging. Development of computational techniques and models for automatic analysis and representation of such natural languages is known as natural language processing (NLP). The base unit of any natural language is a word, and its representation is a challenging task as decoding its actual semantic role is vital for any NLP application. One of the most popular computation models is artificial neural network (ANN). However, with the birth of deep learning, a new era has started in computational linguistic research as representation of words has been redefined in terms of word embeddings which capture words semantics in the form of real-valued vectors. This paper presents lifespan of ANN from discovery of first artificial neuron to current era of deep learning. Further, it follows the journey of word embeddings, analyzes their generation methods along with their objective functions, and concludes with current research gaps.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa: Natural Language Processing (almost) from Scratch. Journal of Machine Learning Research, 12, pp. 2493–2537, (2011)
C. Lala and S. B. Cohen: The Visualization of Change in Word Meaning over Time using Temporal Word Embeddings. CoRR, vol. abs/1410.4 (2014)
P. F. Brown, P. V Desouza, R. L. Mercer, V. J. Della Pietra, and J. C. Lai: Class-based n-gram models of natural language. Comput. Linguist., vol. 18, no. 4, pp. 467–479 (1992)
S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman: Indexing by latent semantic analysis. Journal of the American Society for Information Science, vol. 41, pp. 391–407 (1990)
A. Mnih: Learning word embeddings efficiently with noise-contrastive estimation. Nips, pp. 2265–2273 (2013)
Lebret R, Collobert R.: Word emdeddings through hellinger PCA. arXiv preprint arXiv:1312.5542 (2013)
Pennington J, Socher R, Manning CD. Glove: Global Vectors for Word Representation. EMNLP, Vol. 14, pp. 1532–1543 (2014)
Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin: A Neural Probabilistic Language Model. J. Mach. Learn. Res., vol. 3, pp. 1137–1155 (2003)
R. Collobert and J. Weston: A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. Architecture, vol. 20, no. 1, pp. 160–167 (2008)
T. Mikolov, G. Corrado, K. Chen, and J. Dean: Efficient Estimation of Word Representations in Vector Space. Proc. Int. Conf. Learn. Represent. (ICLR 2013), pp. 112 (2013)
T. Mikolov, K. Chen, G. Corrado, and J. Dean: Distributed Representations of Words and Phrases and their Compositionality. Nips, pp. 1–9 (2013)
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov: Enriching word vectors with subword information. arXiv Prepr. arXiv:1607.04606 (2016)
F. Hill, K. Cho, S. Jean, C. Devin, and Y. Bengio: Not All Neural Embeddings are Born Equal. arXiv, p. 4, (2014)
M. Baroni, G. Dinu, and G. Kruszewski: Dont count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 238–247 (2014)
W. S. McCulloch and W. Pitts: A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys., vol. 5, no. 4, pp. 115–133, (1943)
Hebb, Donald Olding.: The organization of behavior: A neuropsychological approach. John Wiley & Sons, (1949)
F. Rosenblatt: The perceptron: A probabilistic model for information storage and organization in the brain. Psychol. Rev., vol. 65, no. 6, p. 386, (1958)
B. Widrow and M. E. Hoff: Associative Storage and Retrieval of Digital Information in Networks of Adaptive Neurons. in Biological Prototypes and Synthetic Systems, Springer, p. 160, (1962)
Ivakhnenko, A.G. and Lapa, V.G.: Cybernetic predicting devices. (No. TR-EE66-5). Purdue Univ Lafayette Ind School of Electrical Engineering. (1966)
Ivakhnenko, A.G. and Lapa, V.G.: Cybernetics and forecasting techniques. (1967)
Mermelstein, P. and Eden, M.: Experiments on computer recognition of connected handwritten words. Information and Control, 7(2), pp. 255–270. (1964)
J. J. Hopfield: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci., vol. 79, no. 8, pp. 2554–2558, (1982)
T. Kohonen: Self-organized formation of topologically correct feature maps. Biol. Cybern., vol. 43, no. 1, pp. 5969, (1982)
D. E. Rumelhart, G. E. Hinton, and R. J. Williams: Learning internal representation by back propagation. Parallel Distrib. Process. Explor. Microstruct. Cogn., vol. 1, (1986)
M. I. Jordan: Serial order: a parallel distributed approach (ICS Report 8604). San Diego: University of California. Inst. Cogn. Sci., (1986)
D. S. Broomhead and D. Lowe: Radial basis functions, multi-variable functional interpolation and adaptive networks. R. SIGNALS RADAR Establ. MALVERN (UNITED KINGDOM), vol. No. RSRE-M, (1988)
C. Cortes and V. Vapnik: Support-vector networks. Mach. Learn., vol. 20, no. 3, pp. 273–297, (1995)
S. Hochreiter and J. Schmidhuber: Long short-term memory. Neural Comput., vol. 9, no. 8, pp. 1735–1780, (1997)
G. E. Hinton and R. R. Salakhutdinov: Reducing the dimensionality of data with neural networks. Science (80.), vol. 313, no. 5786, pp. 504–507 (2006)
J. Guo, W. Che, H. Wang, and T. Liu: Revisiting Embedding Features for Simple Semi-supervised Learning. ACL pp. 110–120 (2014)
A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov: Bag of tricks for efficient text classification. arXiv Prepr. arXiv:1607.01759 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Soni, V.K., Gopalani, D., Govil, M.C. (2019). Resurgence of Deep Learning: Genesis of Word Embedding. In: Panigrahi, B., Trivedi, M., Mishra, K., Tiwari, S., Singh, P. (eds) Smart Innovations in Communication and Computational Sciences. Advances in Intelligent Systems and Computing, vol 669. Springer, Singapore. https://doi.org/10.1007/978-981-10-8968-8_11
Download citation
DOI: https://doi.org/10.1007/978-981-10-8968-8_11
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-8967-1
Online ISBN: 978-981-10-8968-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)